Market Regimes and Changing Market Dynamics

Identifying Market Regimes with Machine Learning and Entropy

Author

Nam Nguyen
August 09, 2026 • Estimated Reading Time: 5 minutes

Markets have been behaving unusually lately. In May, equity indices rose while volatility and skew also increased, a relatively rare occurrence historically. Since last week, the same phenomenon has emerged again, with the spot/volatility correlation turning positive.

Is this still a rare occurrence? We don't know. But one thing is clear: regime detection is becoming increasingly important in today's markets. In this issue, we explore a couple of approaches for detecting market regimes.

In this issue:

Latest Posts

  • Making Option Pricing Models More Practical (10 min)

  • Algorithmic Trading, HFT, and Market Stability (10 min)

  • The Market Impact of Retail Options Trading (10 min)

  • Why System Validation Matters More Than Ever (10 min)

  • Does Regression Still Work in Modern Markets? (12 min)

Upcoming Conferences

  • The CBOE Quant Conference is a gathering of thought leaders from academia and industry to examine the future of quantitative finance. This year, they feature fantastic speakers such as Emanuel Derman and Fabio Mercurio. If you're a member of an academic institution, you can receive a substantial discount. Follow the link below for more details

  • I will be speaking at the Future Alpha Conference next March. The topic is still being determined, but the discussion will focus on risk management, AI, and machine learning. Stay tuned for more details.

A Regime Classification Framework for Mean-Reverting and Trending Markets

Regime classification is important in asset and risk management. Traditional approaches classify regimes based on direction, bullish or bearish, and volatility, high or low.

Reference [1] departs from this framework and instead classifies markets as mean-reverting or trending. Specifically, it uses return thresholds of 0.5%, 0.75%, and 1% to define regimes and examines SPY, QQQ, DIA, and IWM over the period 2000 to 2024.

Findings

  • The study evaluates Random Forest and Neural Network classifiers using macroeconomic announcement indicators and technical features, including VIX, RSI, and ATR.

  • It uses 25 years of daily data from 2000–2024 for IWM, SPY, QQQ, and DIA.

  • The study frames next-day ETF behavior as a binary classification problem between “oscillating” and “trending” days.

  • Oscillating days are defined using intraday movement thresholds of 0.5%, 0.75%, and 1%, with movements exceeding these thresholds classified as trending.

  • At the 0.5% threshold, Neural Networks outperform a naive classifier by 13.4% for IWM, 15.4% for SPY, 4.7% for QQQ, and 3.2% for DIA.

  • SPY produces the strongest results, with AUC values reaching 0.67–0.74 at the 0.75% and 1% thresholds.

  • IWM shows improvements of 5.7%–13.4% across thresholds, with evidence of predictive power at the 0.5% and 0.75% thresholds.

  • QQQ shows improvements of 4.7%–6.1%, but its predictive performance is weaker at lower thresholds.

  • The results show that predictive performance varies materially across ETFs and oscillation thresholds, with some configurations providing limited discriminatory power.

In summary, the results show that the best case achieves a 15.4% improvement in prediction over a naive strategy for SPY using a neural network with a 0.5% threshold; although in many cases the improvement is more modest, in the range of 1 to 5%, and varies significantly across ETFs.

While the study has several limitations, it points to a more relevant research direction: predicting the magnitude-based regime appears slightly easier than predicting direction, and machine learning is effective as a risk or regime filter rather than as a direct alpha-generating signal.

Reference

[1] Azizi, S. (2026), Leveraging Machine Learning for Financial Forecasting: Distinguishing Market Trends from Oscillations in ETFs, Journal of Risk and Financial Management, 19(4), 262.

Entropy-Based Regime Detection of Tail Risks

Reference [2] proposes an alternative regime classification by distinguishing between “normal” and heavy-tailed regimes. Specifically, the study develops a nonparametric method to detect financial market regimes using differential entropy rather than volatility alone. The underlying idea is that while volatility measures dispersion, entropy captures the full distributional uncertainty, including tail behavior, which becomes particularly important during crisis periods.

The authors estimate entropy using a kernel density estimator with a heavy-tailed kernel in rolling windows and compare entropy with variance. When markets behave approximately Gaussian, i.e., normally, entropy and variance move together; during turbulent periods, the relationship breaks down, revealing heavy-tailed regimes that volatility alone cannot identify.

Findings

  • The study develops a differential entropy approach to identify financial market regimes through changes in distributional complexity rather than variance alone.

  • The method uses a data-adaptive heavy-tailed kernel and combines entropy with tail-index analysis within a moving-window framework.

  • Monte Carlo experiments show that the approach is robust and sensitive to changes in tail behavior.

  • Applied to the Ibovespa, S&P 500, Nikkei, and SSE Composite from 1998 to 2025, the method identifies heavy-tailed regimes associated with major periods of market turbulence.

  • These periods include the Dot-com Bubble, Global Financial Crisis, COVID-19 shock, and the 2025 tariff-related crisis.

  • Gaussian regimes correspond to periods of relative stability and market efficiency.

  • The results show that variance and entropy do not necessarily move together during crises.

  • While volatility measures dispersion, entropy captures broader uncertainty and tail risk, providing a complementary measure of systemic instability.

In short, the paper developed a regime detection method based on entropy, which provides an alternative regime indicator that captures tail risk and structural shifts that standard volatility measures may miss.

This represents an important contribution to the literature, particularly in the context of managing tail risks and risk management more broadly.

Reference

[2] Raul Matsushita, Iuri Nobre, Sergio Da Silva, Beyond volatility: Using differential entropy to detect financial market regimes, Chaos, Solitons and Fractals 202 (2026) 117553

Closing Thoughts

Together, these studies highlight two different approaches to market regime detection. The first uses machine learning to classify next-day ETF behavior as oscillating or trending, while the second uses differential entropy to identify shifts in market uncertainty and tail behavior. Both demonstrate that market regimes can be characterized using information beyond conventional volatility measures, although their effectiveness varies across markets, thresholds, and conditions.

Additional Reading

For further discussion on market regimes, refer to the previous issues:

Educational Video

Regime Switching Models with Machine Learning by Piotr Pomorski

In this video, Piotr Pomorski discusses the importance of regime switching in financial markets, where periods can differ substantially in returns, volatility, and correlations. Traditional Markov-switching regression models can identify whether markets are currently in low- or high-variance regimes, but he argues that the more useful problem for asset management is predicting regime changes in advance. This could allow portfolio managers to adjust allocations before market conditions deteriorate rather than simply recognizing a turbulent regime after it has already begun. 

Pomorski then presents his approach to improving regime identification by combining Markov-switching regression with technical analysis. In particular, he uses Kaufman’s Adaptive Moving Average, which incorporates volatility and trend, to construct four regimes that distinguish not only bullish and bearish markets but also high- and low-variance conditions. These regimes provide labels for subsequent machine-learning models designed to predict regime changes. He also discusses several machine-learning approaches used in finance, including Random Forest, XGBoost, neural networks, unsupervised learning, and reinforcement learning, while emphasizing that limited and costly financial data remain a major practical constraint. 

Volatility Weekly Recap

The figure below shows the term structures for the VIX futures (in colour) and the spot VIX (in grey).

Stocks rallied to fresh all-time highs this week, except for the Nasdaq, as oil prices and Treasury yields declined amid hopes for progress toward a ceasefire with Iran. WTI crude fell 7.7% for the week, helping support equity sentiment.

Technology stocks continued to rebound, supported by positive AI-related developments from Microsoft, Amazon, and Palantir. The S&P 500 also broke out of its three-month trading range to a new all-time high, potentially adding buying pressure from performance chasing and short covering. Second-quarter earnings remained strong. Of the 442 S&P 500 companies that have reported, 69% beat revenue estimates and 87% beat earnings estimates.

On the volatility front, both the spot VIX and VIX futures remained in contango. VIX futures are well above spot, resulting in a firmly positive roll yield. The spot/volatility correlation has turned positive, as shown in the lower panel of the figure below.

Around the Quantosphere

  • More than 5,000 investors are still copying Leopold Aschenbrenner's trades despite his hedge fund's meltdown (finance.yahoo)

  • July Was the Month the Crowded AI Trade Reversed (institutionalinvestor)

  • The Quantamental Advantage in Fixed Income Investing (troweprice)

  • Wall Street's 'Fear Gauge' Is Doing Something Unusual (cnbc)

  • Investors in Situational Awareness Deserved to Lose Their Shirts (economist, paywall)

  • Dating insights from a hedge fund quant: Sample, calibrate, "act decisively when you see something better" (efinancialcareers)

  • Hedge Funds Build Biggest Bearish Canadian Dollar Position in Two Years (hedgeweek)

  • AI Is Turning Retail Traders Into DIY Hedge Funds (bloomberg, paywall)

Disclaimer

This newsletter is not investment advice. It is provided solely for entertainment and educational purposes. Always consult a financial professional before making any investment decisions.

We are not responsible for any outcomes arising from the use of the content and codes provided in the outbound links. By continuing to read this newsletter, you acknowledge and agree to this disclaimer.