Does Regression Still Work in Modern Markets?

Regression in the Age of Machine Learning and AI

Author

Nam Nguyen
May 31, 2026 • Estimated Reading Time: 6 minutes

Regression is one of the oldest and widely used statistical techniques. It has found applications across the social sciences, engineering, natural sciences, and finance. Despite the rapid rise of machine learning and AI, regression remains a useful tool for modeling relationships, making forecasts, and extracting signals from data.

In this edition, we revisit regression-based trading systems and examine whether simple linear and logistic regression models can still generate useful predictive signals in today's increasingly complex financial markets.

In this issue:

Latest Posts

  • Volatility Derivatives and VIX Market Dynamics (10 min)

  • Overfitting and Parameter Selection in Trading Strategies (10 min)

  • Volatility Risk Premium and Clustering: Intraday vs Overnight Dynamics (8 min)

  • Large Language Models in Trading: Models and Market Dynamics (9 min)

  • Evaluating Option-Based Strategies and Dollar-Cost Averaging (10 min)

Is Linear Regression Still a Good Prediction Method?

Forecasting stock prices is a challenge due to the non-stationary nature of price time series and the noisy data inherent in these price sequences. Linear regression was a frequently used prediction method, but recent advancements in computing technologies have given rise to more sophisticated approaches like Long Short-Term Memory (LSTM), Artificial Neural Networks (ANN), Recurrent Neural Networks (RNN), etc.

Does the linear regression method still have its place amongst these advanced techniques?

Reference [1] examines the effectiveness of the linear regression method by applying it to a set of US stocks, using it for predicting closing prices and 10-day moving averages.

Findings

  • The study develops a stock prediction framework based on historical prices, economic indicators, and linear regression techniques.

  • The authors construct two models: one for stock price forecasting and another for predicting the 10-day Exponential Moving Average (EMA_10).

  • The methodology includes data cleaning, feature selection, model training using Ordinary Least Squares (OLS), and performance evaluation using RMSE and MAE metrics.

  • Both models achieve low prediction errors and high explanatory power, as reflected by favorable RMSE, MAE, and R-squared statistics.

  • The results suggest that the models provide accurate forecasts of stock prices and short-term trend indicators.

  • The proposed trading strategy generates profitable results while also reducing portfolio risk.

  • The study concludes that simple linear regression models can provide useful insights into future stock price movements and market trends.

In summary, linear regression is still an effective prediction method. It remains a viable method due to its

  • Simplicity and interpretability,

  • Efficiency with smaller datasets,

  • Ability to mitigate excessive overfitting.

Reference

[1] S. Sanapala, V. A. Reddy, S. Sinha Choudhury, V. V. Akshaya and V. Maheedhar Varma, Optimising Trading Strategies using Linear Regression on Stock Prices, 2023 International Conference on Research Methodologies in Knowledge Management, Artificial Intelligence and Telecommunication Engineering (RMKMATE), Chennai, India, 2023, pp. 1-6.

Evaluating a Logistic Regression Trading Framework

Reference [2] employs logistic regression, which is particularly suited for modeling binary outcomes, to predict stock price movements based on historical returns.

The author uses cumulative returns over the past 20 days and the past 12 months as predictive variables, capturing short-term and long-term momentum effects. Logistic regression is then applied to classify whether a stock’s return in the upcoming month exceeds that month’s median return. The procedure is implemented on S&P 500 stocks from January 1985 to July 2024 using survivorship-bias-free data.

Findings

  • The paper evaluates a Logistic Regression-Based Systematic Trading (LRST) strategy applied to S&P 500 stocks from 1983 to 2023.

  • The strategy uses logistic regression to predict future stock price direction based on historical returns and frames the problem as a binary classification task.

  • The model employs a rolling 10-year estimation window, allowing it to adapt to changing market conditions over time.

  • Over the full sample, the strategy achieves an annualized return of 24.61%, outperforming the S&P 500 during several periods, particularly in the 1990s and early 2000s.

  • Despite strong historical returns, the strategy exhibits substantial risk, with an annualized volatility of 26.11% and a Sharpe ratio of 0.77.

  • Recent performance from 2021 to 2024 is notably weak, with the strategy failing to participate in much of the market's gains.

  • The results suggest that structural market changes, including the growth of algorithmic trading and shifting macroeconomic conditions, may have reduced the strategy's effectiveness.

  • The study highlights the importance of adapting systematic trading models to evolving market environments.

  • The authors suggest that incorporating machine learning methods, sentiment indicators, and macroeconomic variables could improve robustness and future performance.

In short, the paper shows that the logistic regression-based strategy delivers an annualized return of 24.61%, outperforming the S&P 500, but its high volatility and Sharpe ratio of 0.77 indicate substantial risk and room for improvement in its risk-return profile. Its recent underperformance may reflect structural weaknesses amid the rise of algorithmic trading and shifting macroeconomic conditions, underscoring the need for adaptation.

This article is insightful as it demonstrates that,

  • Even a basic regression framework can serve as a useful predictive tool within a trading system, although further refinement is necessary.

  • There might be structural changes in market dynamics, driven by the increasing prevalence of algorithmic trading and artificial intelligence, implying that traders must adapt accordingly.

Reference

Closing Thoughts

Taken together, these studies suggest that simple regression techniques, whether linear or logistic, remain useful tools for systematic trading even in the modern era. Despite the rapid growth of machine learning and AI, relatively straightforward models can still generate meaningful predictive signals and attractive historical performance.

However, the papers also highlight that refinement is necessary, as the effectiveness of these models depends on market conditions, structural changes, and the choice of predictive variables. Continuous adaptation and model improvement remain essential for maintaining performance over time.

Additional Reading

For further discussion on the regression technique in trading, please refer to the previous issue:

Use of Machine Learning in Pairs Trading (in Simplicity or Complexity? Rethinking Trading Models in the Age of AI and Machine Learning)

Educational Video

MIT Lecture:  Regression Analysis in Finance by Dr. Peter Kempthorne

In this video, Dr. Peter Kempthorne provides an accessible introduction to linear regression, one of the most widely used statistical tools in finance and many other disciplines. He explains that regression can be used for prediction, causal analysis, approximation, and uncovering relationships between variables. The lecture develops the mathematical foundations of multiple linear regression, including the specification of dependent and explanatory variables, residual errors, and the Ordinary Least Squares (OLS) framework. A key takeaway is that regression is far more flexible than it first appears, as nonlinear relationships can often be incorporated through transformations, polynomial terms, Fourier series, and time-series extensions.

The lecture also emphasizes that building a successful regression model requires much more than estimating coefficients. Model assumptions must be carefully checked, residuals analyzed, and the specification refined when necessary. Dr. Kempthorne discusses the Gauss-Markov theorem, which shows that OLS provides the best linear unbiased estimator under certain assumptions, and explains how the framework can be extended to accommodate correlated errors, unequal variances, and non-normal distributions. A recurring theme throughout the lecture is that statistical modeling is an iterative process: rather than applying regression mechanically, practitioners should adapt the model to the characteristics of the data and the underlying process being studied. 

Volatility Weekly Recap

The figure below shows the term structures for the VIX futures (in colour) and the spot VIX (in grey).

Stocks advanced during the holiday-shortened week, with the S&P 500 and Nasdaq reaching new highs as strong technology earnings and continued enthusiasm around AI supported risk appetite. However, gains were tempered by persistent inflation concerns and ongoing uncertainty surrounding the conflict in Iran. Non-energy minerals, consumer durables, and technology stocks led the market higher, while energy, industrial services, and utilities lagged.

Outside equities, markets remained focused on inflation and geopolitics. Treasury yields stayed elevated as investors increased the probability of a Fed rate hike later this year. Oil prices remained volatile, reacting to developments in the Middle East and concerns over supply disruptions. Gold struggled as rising rate expectations offset its safe-haven appeal, while Bitcoin and the broader crypto market moved lower as inflation and geopolitical risks weighed on sentiment.

In the volatility market, both spot VIX and VIX futures remain firmly in contango. Roll yield is positive and continues to increase. As a result, VXX has continued to benefit from the favorable term structure, catching up with the S&P 500 and establishing a new low.

Around the Quantosphere

  • Bank of Canada says financial system is in good shape, but vulnerabilities have increased (cbc)

  • Beyond the Data: Hedge Fund Algorithmic Usage Shifting From Optimal to Essential (thetradenews)

  • Hedge fund superstar does due diligence on his new boss. People are getting protective of their clients (efinancialcareers)

  • Prediction Markets Target Hedge Funds for Next Growth Push (hedgeweek)

  • Rising hedge fund leverage affects monetary policy implementation (dallasfed)

  • Robinhood Is Letting AI Trade for You So You Don’t Have to Keep Checking the Markets (coindesk)

  • AI Is Making Even Tech Hedge Funds Look Like the S&P 500 (institutionalinvestor)

  • Hedge Funds Are Losing Their Edge in a World of ETFs? (advisorperspectives)

  • The Evolution of Quant Finance: A Conversation With Bruno Dupire, 2025 IAQF/Northfield Financial Engineer of the Year (bloomberg)

  • The billionaire hedge fund manager whose wife disagreed with self-enrichment. The redemption of Morgan Stanley's tech team (efinancialcareers-canada)

  • The Quiet Revolution in Trading: Why the Future May Reward Patience More Than Prediction (globalbankingandfinance)

Disclaimer

This newsletter is not investment advice. It is provided solely for entertainment and educational purposes. Always consult a financial professional before making any investment decisions.

We are not responsible for any outcomes arising from the use of the content and codes provided in the outbound links. By continuing to read this newsletter, you acknowledge and agree to this disclaimer.