An exploratory project applying PCA and Sparse PCA to 2-minute returns for 457 S&P 500 stocks over 28 trading days (August 12 to September 19, 2024). Revised in October 2026 with corrections and a note on what I would do differently.
I wrote this post in October 2024, shortly after finishing my M.S., as a hands-on way to learn PCA and Sparse PCA on real market data. Re-reading it two years later, the data pipeline and the plots hold up, but much of the original interpretation does not. I read macro and geopolitical stories into six weeks of 2-minute returns, and some of the methods had problems I did not catch at the time.
In October 2026 I revised the text. The code and data are unchanged, and so is every figure except the rolling-window dashboard, where I fixed a labeling bug (see below). The edits are to the writing, plus a few quick checks against the outputs saved in the project repository; nothing was re-run. The original title was “Detecting Regime Shifts in SP500 Stocks Using PCA and Sparse PCA”. I changed it because the project explores structure; it does not detect regime shifts.
What changed in this revision:
What I would do differently today:
This project applies Principal Component Analysis (PCA) and Sparse PCA to high-frequency returns of S&P 500 stocks. The goal was exploratory. I wanted to see:
PCA is a standard tool for this kind of data. It compresses hundreds of correlated return series into a handful of components, the first of which usually tracks the market as a whole. Sparse PCA adds a penalty that pushes many loadings to zero, which can make components easier to read.
The raw download covers all 503 S&P 500 constituents and 20 ETFs at 2-minute intervals from the Yahoo Finance API. After cleaning, the sample is 457 stocks over 28 full trading days, from August 12 to September 19, 2024.
The project was also a way for me to practice end-to-end machine learning work on real data: acquisition, cleaning, modeling and interactive visualization. It is best read as an exploration, not as a set of trading conclusions.
The download included all 503 S&P 500 stocks and 20 ETFs at 2-minute intervals. Yahoo Finance only provides 2-minute bars for roughly the last 60 days, which limited the sample length.
I kept regular market hours only, from 9:30 AM to 4:00 PM ET. This gives 195 bars per trading day and avoids the thin, irregular trading of the pre-market and after-hours sessions.
The data cleaning process involved the following steps:
Handling Missing Data: I filled short gaps forward and then backward, up to 5 intervals, and dropped any rows that were still incomplete. In hindsight this has two side effects. Backward filling uses later prices to fill earlier gaps, and every filled price creates an artificial zero return. With a 5-interval limit the effect is probably small, but today I would mask these points instead.
Asset Filtering: I removed any asset with more than 20% missing data. This kept 457 of the 503 stocks.
Timestamp Synchronization: All series were aligned to a common time index, so that price movements are compared at the same moments across the dataset.
Return Calculation: I worked with logarithmic returns rather than prices. Log returns are additive over time and closely approximate percentage changes over short intervals. Because the returns were computed on the stitched-together series, the first return of each day spans the overnight gap (see the note at the top).
Standardization: I standardized each stock’s return series to mean 0 and standard deviation 1. This keeps stocks with different volatilities on a comparable scale.
The cleaned dataset has 5,463 synchronized timestamps, about 97% of the 5,655 in the raw download. These give 5,462 2-minute returns for 457 stocks and 20 ETFs. A handful of stocks have no data on the first day of the download (August 9), so dropping incomplete rows removes almost all of that day. The analyzed sample therefore effectively runs from August 12 to September 19.
PCA transforms a dataset into a new coordinate system whose axes, the principal components, are ordered by how much variance they explain. This section covers three steps:
I applied PCA to the full sample of 457 stocks. PCA finds a set of orthogonal directions (the principal components) that maximize the variance of the projected data.
For a data matrix $X$, PCA solves the eigenvalue problem:
\[(X^TX)v_i = \lambda_i v_i\]where $v_i$ are the eigenvectors (principal components) and $\lambda_i$ are the corresponding eigenvalues, which measure the variance explained by each component.
Components are ordered by decreasing explained variance, so PC1 explains the most, PC2 the second most, and so on.
I used sklearn’s PCA implementation and kept enough components to explain 90% of the variance.
The cumulative explained variance plot shows how quickly variance is captured. The first component alone explains about 24.5% of the variance and the second about 7.4%. After that the curve flattens: 154 components are needed for 80% of the variance, and 255 (slightly more than half of the 457) for 90%.
That long tail is expected at this frequency. Over 2-minute intervals, much of each stock’s return is idiosyncratic noise and market microstructure (bid-ask bounce, asynchronous trading). Correlations between stocks are also weaker than at daily frequency, a pattern known as the Epps effect. So one large market component followed by a long, flat tail is the typical picture, and many of the trailing components are likely indistinguishable from noise (see the note at the top).
PC1 has a positive loading on every stock and behaves like a market factor. Beyond the first few components, principal components usually mix several influences. They should not be read as clean economic factors without further work.
The second plot is a heatmap of each stock’s loadings on the first 10 principal components, with stocks grouped by clustering. Groups of stocks with similar loadings are visible, and I suspected many of them correspond to sectors. Sparse PCA seemed a natural next step for making that structure easier to read.
Standard PCA components involve every input variable, which makes them hard to interpret. Sparse PCA adds a penalty that pushes many loadings to exactly zero, so that each component is described by fewer stocks.
In sklearn’s formulation, Sparse PCA solves:
\[\min_{U,V} \frac{1}{2}\lVert X - UV \rVert_F^2 + \alpha \lVert V \rVert_{1} \quad \text{subject to } \lVert U_k \rVert_2 \le 1 \text{ for all } k\]Here the rows of $V$ are the components, that is, the loadings over stocks, which are the part being made sparse. $U$ holds the corresponding component scores over time, and $\alpha$ controls the strength of the sparsity penalty.
I used sklearn’s SparsePCA with 10 components and the default penalty, $\alpha = 1$. Two things about the result matter for everything that follows:
In principle, Sparse PCA trades some explained variance for interpretability. The main practical drawbacks are:
To see how sectors contribute to the Sparse PCA components, I mapped each stock to a sector using the yfinance API and merged the classifications into ten categories:
For each component, I summed the positive loadings by sector and divided by the component’s total positive loading, and did the same for the negative loadings. The percentages below are each sector’s share of a component’s positive or negative side.
One thing I got wrong in the original version: the sign of a principal component is arbitrary. Multiplying a component by -1 gives an equally valid solution. So “positive” and “negative” only tell you which groups of stocks move against each other within a component. They do not tell you which sector helps or hurts. Some of my original readings do not survive this, such as treating technology’s frequent appearance on the negative side as evidence of its role as a “disruptive force”.
The dominant sectors on each side of each component:
| PC | Positive side | Negative side |
|---|---|---|
| 1 | Industrials (36.6%) | Consumer (41.8%), Healthcare (21.8%) |
| 2 | Utilities (51.1%), Consumer (19.2%) | Technology (37.3%) |
| 3 | Energy (45.1%) | Consumer (35.5%), Technology (26.4%) |
| 4 | Consumer (51.1%) | Technology (28%), Consumer (16.7%) |
| 5 | Real Estate (50%) | Consumer (43.2%), Technology (18.1%) |
| 6 | Consumer (34.3%), Industrials (18.5%) | Healthcare (22.9%), Financial Services (20.8%) |
| 7 | Financial Services (57.6%) | Healthcare (21.8%), Technology (20.6%), Consumer (17.4%) |
| 8 | Consumer (30.4%), Healthcare (28.2%) | Industrials (22.3%), Technology (18.6%) |
| 9 | Technology (55%) | Financial Services (25.4%), Consumer (23.4%), Healthcare (20%) |
| 10 | Healthcare (34.7%), Consumer (25.7%) | Consumer (31.7%), Financial Services (17.1%), Technology (13.6%) |
What I think the table does show:
Two caveats apply. Because the components are only mildly sparse, each sector share is spread over many stocks, and the shares describe tilts rather than clean sector factors. These results also come from a single six-week window and may look different in another period.
To connect the components to something observable, I correlated each component’s scores with the returns of 20 ETFs:
Component Scores: I projected the standardized stock returns onto the PCA and Sparse PCA loadings to get a time series of scores for each component.
Correlation Analysis: I computed Pearson correlations between each component’s scores and each ETF’s standardized returns.
Visualization: The correlations are shown as heatmaps.
Lag Analysis: To look for leading or lagging relationships, I repeated the correlations with the ETF returns shifted by up to 5 intervals (10 minutes) in either direction.
Lead-lag relationships were weak at shifts of up to 10 minutes:
This is roughly what you would expect. Arbitrage keeps liquid ETFs in line with their constituents, so most new information shows up in both within the same 2-minute bar. The small one-bar ETF lead is consistent with some individual stocks’ 2-minute prices being slightly stale relative to the more heavily traded ETFs. The lag analysis plots are available here.
Two things make the high correlations below less surprising than they look:
Full PCA Results:
| PC | Strong Positive Correlations | Strong Negative Correlations |
|---|---|---|
| PC1 | XLI (0.929), DIA (0.917), XLB (0.905), XLF (0.856), IWM (0.850) | VXX (-0.608) |
Sparse PCA Results:
| PC | Strong Positive Correlations | Strong Negative Correlations |
|---|---|---|
| PC1 | XLI (0.946), XLB (0.879), DIA (0.874), XLF (0.819), IWM (0.806) | VXX (-0.590) |
| PC2 | XLU (0.909) | XLK (-0.158), VXX (-0.131) |
| PC3 | XLE (0.944), XLB (0.823) | VXX (-0.416) |
| PC4 | VXX (0.684) | SPY (-0.862), DIA (-0.835), XLC (-0.816), XLI (-0.807), XLY (-0.807) |
| PC5 | XLRE (0.917), DIA (0.731), XLI (0.727), XLF (0.722) | VXX (-0.444) |
| PC6 | XLI (0.890), XLB (0.884), IWM (0.882), DIA (0.868) | VXX (-0.570) |
| PC7 | XLF (0.942), DIA (0.883), XLI (0.868), XLB (0.843) | VXX (-0.525) |
| PC8 | XLV (0.827), XLP (0.824), DIA (0.757), XLF (0.754) | VXX (-0.338) |
| PC9 | QQQ (0.938), XLK (0.918), SPY (0.910) | VXX (-0.728) |
| PC10 | IWM (0.590), DIA (0.588), XLV (0.577), XLI (0.562) | VXX (-0.406) |
Overall, the ETF correlations confirm what the sector shares suggested: one dominant market factor, plus components that are mostly the market seen through a particular sector. That is a useful sanity check that the components pick up real structure. On its own, though, it says little about macro drivers.
So far everything has been static, treating the whole sample as one period. The sample includes the Federal Reserve’s 50 basis point rate cut on September 18, 2024, so the natural next question was whether the structure visibly changes over time.
To look at how the structure evolves, I repeated the analysis on rolling windows.
I applied both standard PCA and Sparse PCA using rolling windows with the following parameters:
This gives 19 windows. Each is labeled by its last day, from August 23 to September 19. Consecutive windows share nine of their ten days, so they are expected to look similar.
For each window, I:
Both metrics are computed from the standard PCA results. For each of the first 10 components, I took the sector shares of its positive loadings (the same construction as in the intraday section below) and flattened them into a single vector.
Market Stability compares consecutive windows using cosine similarity:
\[\mathrm{Stability}_{i, i+1} = 1 - \mathrm{cosine\_distance}(\mathrm{loadings}_i, \mathrm{loadings}_{i+1})\]where $\mathrm{loadings}$ are the flattened sector-share vectors. Higher values mean consecutive windows look more alike.
Similarity to the Full Period compares each window with the full-sample result:
\[\mathrm{SimFull}_i = 1 - \mathrm{cosine\_distance}(\mathrm{loadings}_i, \mathrm{loadings}_{\mathrm{full}})\]In the dashboard this panel is labeled “Deviation from Full Period”. Despite that name, it is a similarity score: lower values mean the window looks less like the full sample.
The interactive dashboard combines four panels:
A slider at the bottom moves through the time windows.
Stepping through the windows shows a few patterns. In the original version of the dashboard, the sector labels stopped matching the rows once the slider moved, and my original observations were read off those wrong labels. The figure is now fixed, and the observations below were re-read from the underlying data:
Consumer early on: In every window ending August 23 to September 4, consumer stocks lead half or more of the ten standard-PCA components. Afterwards they lead three or fewer. Consumer is also the largest of my merged sectors, so it tends to lead whenever loadings are spread evenly. This is partly a sector-size effect.
Technology and utilities in PC2: Utilities lead standard PC2 in the first five windows (ending August 23 to 29) and again in the last one. Technology leads it in every window from August 30 to September 18 except September 3, at 44-54%. In Sparse PCA, PC2 mostly alternates between the same two sectors.
Energy in PC3: Energy leads PC3 in both methods from late August to mid-September, at 32-49%. In standard PCA that is the windows ending August 30 to September 12; in Sparse PCA, August 29 to September 16.
Financial Services in PC1: Financial services leads Sparse PC1 in every window ending August 30 to September 16. It also leads standard PC1 in the windows ending September 11 to 17, but only with 17-18%. Standard PC1 is the market factor, so its loadings are spread across all sectors.
Real Estate and Healthcare later: Real estate leads PC4 in both methods in the windows ending September 13 to 19. From early September, healthcare leads one or more of the later components in most windows.
These observations are descriptive. The sign and order of components can change from one window to the next. Some of these shifts may therefore reflect a component flipping sign or swapping places with another, rather than a real change in market structure.
The stability values range from 0.76 to 0.97, and most are above 0.8.
In the original version I matched each dip to a news event, such as a producer-price inflation release on September 12 and the Fed’s decision on September 18. I no longer think the metric supports that, for two reasons:
A quick check against the saved loadings makes both points concrete:
After sign alignment, the lowest values are at the steps into the windows ending September 18 and 19, which add the day of the Fed’s 50 basis point cut (WSJ, Sept. 18, 2024) and the day after. August 30 is nearly as low, though. Without a null distribution, I would not call any of this a regime shift.
Similarity to the full sample generally declines over the period. It ranges from about 0.69 to 0.85 in the windows ending before September 13, and from 0.54 to 0.59 in the windows ending September 13 to 19. The window most similar to the full sample is the one ending September 3 (0.849).
One possible explanation is that the last week of the sample, which contains the Fed decision, differs from the rest. However, the same caveats about sign flips and the missing baseline apply, so I treat this as a descriptive observation rather than a finding.
The rolling-window analysis is a useful way to visualize how loadings move around over a short sample. In its current form it is not a regime-shift detector. Making it one would need component alignment across windows and a null distribution for the metrics. The result also depends on the window length, and with heavily overlapping windows, consecutive values are not independent.
Next I looked at whether sector structure differs by time of day. Intraday patterns such as heavier trading at the open and close are well documented, so it seemed worth checking whether they show up in the components.
Data Segmentation: I divided the trading day into 13 periods of 30 minutes each:
\[p_i = [t_i, t_{i+1}), \quad i = 1, 2, ..., 13\]where $t_1$ is the market open (9:30 AM) and $t_{14}$ the market close (4:00 PM).
Sparse PCA Application: For each period $p_i$, I pooled the returns from that half hour across all days and fitted Sparse PCA with 10 components:
\[X_{p_i} \approx U_{p_i} V_{p_i}\]where $X_{p_i}$ is the data matrix for period $p_i$, the rows of $V_{p_i}$ are the sparse loadings, and $U_{p_i}$ holds the component scores.
Sector Loading Calculation: For each component $j$ and sector $S$ in period $p_i$, the sector loading $SL_{S,j,p_i}$ is:
\[SL_{S,j,p_i} = \sum_{k \in S} \max(L_{k,j,p_i}, 0)\]where $L_{k,j,p_i}$ is the loading of stock $k$ in component $j$ for period $p_i$. These were then normalized across sectors:
\[NSL_{S,j,p_i} = \frac{SL_{S,j,p_i}}{\sum_{S'} SL_{S',j,p_i}}\]Visualization: I produced two charts:
a sector influence plot showing each sector’s average influence:
\[ASI_{S,p_i} = \frac{\sum_{j=1}^{10} NSL_{S,j,p_i}}{\sum_{S'} \sum_{j=1}^{10} NSL_{S',j,p_i}}\]Caveats (added in 2026). Four problems limit what this section can show:
I would treat the figures below as a visualization of noisy fits rather than evidence of intraday patterns.
The heatmap shows the sector shares of each Sparse PCA component in each period:
What the heatmap shows:
Changing leaders: The sector that leads a given component changes across periods. For example, financial services leads PC1 at the open (09:30-10:00) and PC7 by mid-morning (10:30-11:00).
Technology: Technology leads PC2 at the open (09:30-10:00) and PC3 in the next period (10:00-10:30). It keeps reappearing later as the leader of PC1, PC2 or PC4 at various points in the day.
Energy: Energy leads PC3 in 8 of the 13 periods.
Utilities: Utilities lead PC5 in about half the periods, mostly in the late morning and the last hour. They also lead PC2 in three periods.
Industrials: Industrials lead PC1 in 7 of the 13 periods.
Consumer: Consumer leads different components at different times.
Last period: In the final period (15:30-16:00), technology again leads PC2, and financial services leads PC3 and PC8.
Because the component numbers come from separate fits, a sector moving from “PC1” to “PC7” does not necessarily mean anything changed. The more consistent patterns, industrials in PC1, energy in PC3 and utilities in PC5, are the ones I would check first if this were redone with more data.
The line plot shows each sector’s overall influence by period:
Consumer: Consumer has the largest swings, peaking at 32.54% in the 11:00-11:30 period.
Technology: Technology starts at 16.90% in the first period, stays between about 9% and 13% for most of the day, and ends at 15.37% in the last period.
Financial Services: Financial services is highest at 20.39% (15:00-15:30) and 18.13% (14:00-14:30), and lowest at 6.70% (11:00-11:30).
Energy: Energy stays between 4.79% and 8.76% throughout the day.
Utilities: Utilities peak at 18.15% in the 14:00-14:30 period.
Communication Services: Communication services switches between two levels: about 3-4% in most periods and 11-15% in 11:00-11:30 and from 12:00 to 14:00, peaking at 14.93% in 13:30-14:00. Jumps between two fixed levels like this are what you would expect if a component led by communication-services stocks enters or leaves the positive side of a fit, rather than if the sector’s influence changed gradually.
In the original version I attached explanations to these movements, such as consumer news releases, daily energy consumption patterns and end-of-day rebalancing. Given the sample-size problem described above, differences of this size between periods could easily come from noise in the fits. I have no test that says otherwise, so I removed those explanations.
The main limitations of this section:
Sample Size per Fit: Each period’s fit has fewer observations than stocks, which is the most serious limitation and is described above.
Overnight Gap: The opening period mixes overnight moves with ordinary 2-minute returns.
Time Period Specificity: The sample covers six weeks in August and September 2024 and may not generalize to other periods.
Aggregation Effects: Pooling returns across days hides day-specific patterns, including responses to scheduled announcements.
Sector Classification: The merged sector categories, especially the combined consumer category, are coarse.
A more convincing version of this analysis would:
What I think holds up:
What does not hold up:
The most useful thing this project taught me was how easy it is to over-read PCA on market data. If I return to it, the list in the note at the top is where I would start.
Project Code: Kal Parvanov/Github