Chapter 4 Experiment Results
4.2 Results
In the experiment, we roll the window to test whether index-tracking portfolio could track the index or the active funds. We hold the portfolio for 22/66 trading days (about a month/a quarter) in the testing period and then rebalance the portfolio in the end of testing period. As shown in figure 4.2, the rolling window will keep moving forward. In the last rolling window, if testing period is less than 22/66 trading days, the previous testing period will extend to the end of data date. Finally, we concatenate testing period in each window together and compare results with the tracked index.
Fig. 4.2 Rolling Window
The optimal numbers of stocks 𝑛∗ in portfolio are different among indexes and active funds. The optimal number in S&P 500, NASDAQ Composite, FSCSX, FBSOX and NASDX index-tracking portfolios is 500, 100, 100, 60 and 50 respectively. The figure 4.3 and the figure 4.4 show cumulative returns of index-tracking portfolio with different testing period. We can see that testing period affect tracking ability a lot. GSPC (S&P 500), IXIC (NASDAX Composite) and NASDX have better tracking ability with 22 testing period. On the other hand, FSCSX and FBSOX have better tracking ability with 66 testing period. However, no matter which testing period we choose, all of the index-tracking portfolio excellently track the indexes and active funds. That is a surprising result since we select stocks from the whole U.S. equity market which consists of 3000~5000 common stocks (exclude ADRs, REITs and closed-end funds) in each period.
Fig. 4.3 Cumulative Returns of Index-Tracking Portfolio and Index (Testing Period: 22 Trading Days)
Fig. 4.4 Cumulative Returns of Index-Tracking Portfolio and Index (Testing Period: 66 Trading Days)
We further compare GSPC and IXIC index-tracking portfolios with ETFs.2 SPDR S&P 500 ETF Trust (SPY) and Fidelity NASDAQ Comp. Index Trk Stk (ONEQ) are the index-tracking ETFs of S&P500 and NASDAX Composite respectively. SPY and ONEQ are the most actively traded index-tracking ETFs and they perfectly track the indexes. The figure 4.5 presents the distribution of tracking difference of index-tracking portfolios and ETFs. All distributions of tracking difference of index-tracking portfolios are bell-shape and centered around zero. The distribution of the index-tracking portfolio with 22 testing period is almost the same as the portfolio with 66 testing period. For GSPC, ETF fits
Fig. 4.5 Distribution of Tracking Difference
2 There is no ETFs for active funds such as FSCSX, FBSOX and NASDX.
better with the index since its distribution of tracking difference has a higher peak and is more centered around zero. For IXIC, the distribution of ETF has only a little difference with the distribution of index-tracking portfolio. The tracking ability may be indifferent according to their distribution.
Our main measurement of index-tracking problem is mean square of tracking difference.
Table 4.2 and table 4.3 give a brief summary of tracking difference year by year. In all panels of both tables, the standard deviation of tracking difference in 2008 and 2009 is relative larger than the tracking difference in other years. Besides, the minimum is smaller and the maximum is larger. We can infer that the index-tracing portfolio built by the deep reinforcement learning model do a bad job in 2008 and 2009. One of the possible reason is that the variability of the whole market is larger due to financial crisis which makes RL agents hard to learn. The other reason is that the original tracking difference expanded in 2008 and 2009 because of larger variability.
From the both tables, we can observe that the standard deviation of tracking difference of portfolios of actives funds, NASDX, FSCSX and FBSOX, are larger than index-tracking portfolios of indexes, S&P500 and NASDAQ Composite, in almost all years. Minimum and maximum also exhibit more deviated from zeros. The major reason is that active funds pay dividends while the index-tracking portfolio don’t. The active funds will have a price gap on ex-dividend date which makes tracking difference become larger.
Table 4.2 Tracking Difference Description (Testing period: 22 Trading Days)
Table 4.2 gives the description of tracking difference. Std is the standard deviation of tracking difference. 25%, 50% and 75% are the first, the second and the third quartile respectively.
Year Mean Std Min 25% 50% 75% Max
2017 -0.0137% 0.1424% -0.7133% -0.1005% -0.0014% 0.0592% 0.6758%
Table 4.3 Tracking Difference Description (Testing period: 66 Trading Days)
Table 4.3 gives the description of tracking difference. Std is the standard deviation of tracking difference. 25%, 50% and 75% are the first, the second and the third quartile respectively.
Year Mean Std Min 25% 50% 75% Max
Year Mean Std Min 25% 50% 75% Max
The deep reinforcement learning model will minimize the mean square of tracking difference (MSTD). We expect low MSTD in the end. However, the question is how low is enough for an index-tracking problem. In order to have a concrete understanding about MSTD, we introduce several measurements, Correlation, R Square, Beta and Tracking Error3, to compare with MSTD. Returns of the index-tracking portfolio and returns of the index are used to calculated the above measurements. Beta is the regression beta coefficient without interception. If Beta is closer to 1, returns of index-tracking portfolio and returns of the index will have more consistent movement, in other words, the tracking ability will be better. Tracking error is the standard deviation of tracking difference. While tracking difference measures the extent to which an index return differs from that of its benchmark index, tracking error indicates how much variability exists among the individual data points that make up the index-tracking portfolio’s average tracking difference. Table 4.4 and table 4.5 present the results of tracking measurements. Note that no matter which indexes or active funds we track, the MSTD in 2008 and 2009 is higher.
Besides, MSTD of indexes is lower than MSTD of active funds. The result coincides with the results we observe from cumulative returns in figure 4.3 and figure 4.4 and deviation of maximum or minimum of tracking difference in table 4.2 and table 4.3. However, we do not get consistent results from other tracking measurements. For indexes which considered to have a better fit, when MSTD and tracking error reach to their maximum in 2008 and 2009, correlation and R square do not seem to be the lowest. Beta doesn’t have a much deviation from one either. Active funds also present the same situation. One
3 Tracking Error = 𝑠𝑡𝑑(𝑟B− 𝑟)) which is standard deviation of tracking difference.
of the possible reason is that though MSTD and tracking error reach to their maximum in 2008 and 2009, the direction of movement of index-tracking portfolios and the indexes are similar in the period of financial crisis. As a result, correlation and R square do not decline much. Beta will not change much either. For an investor, we not only consider the scale of deviation from the tracked indexes but also the direction of movement, which give us room for improvement.
For indexes in panel A and panel B and NASDX in panel C, we observe that MSTD and Tracking Error are lower, Correlation and R square are higher, and beta is closer to one in all years with 66 testing period. It indicates that when the testing period is 66 trading days, the tracking ability of index-tracking portfolio is better. This is totally different from the cumulative return we’ve observed. For active funds in panel D and panel E, MSTD and Tracking Error are lower, Correlation and R square are higher, and beta is closer to one in all years with 22 testing period. It indicates that when the testing period is 22 trading days, the tracking ability of index-tracking portfolio is better. This is totally different from the cumulative return either. The similar situation occurs in active funds.
One of the possible reason is that the positive and negative part of tracking difference offset each other, so the discrepancy of cumulative returns between index-tracking portfolio and target becomes smaller. However, the tracking difference itself may be large, which leads to bad results in all tracking measurements. The tracking measurements indeed give us more insight which we cannot observe in cumulative return.
Table 4.4 Tracking Measurement (Testing Period: 22 Trading Days)
Table 4.4 gives the tracking measurement of the index-tracking portfolio with 22 testing period. Correlation means the correlation between returns of index-tracking portfolio and return of the index. We regress the returns of the index-tracking portfolio on returns of index without interception to get R square and Beta. Tracking error is the standard deviation of tracking difference. MSTD is mean square of tracking difference. In Year column, ‘all’ stands for years from 2006 to 2017.
Year Correlation R square Beta Tracking Error MSTD
Panel A: GSPC
2012 0.9800 0.9605 0.9597 1.92E-03 3.68E-06
Table 4.5 Tracking Measurement (Testing Period: 66 Trading Days)
Table 4.5 gives the tracking measurement of the index-tracking portfolio with 66 testing period.
Correlation means the correlation between returns of index-tracking portfolio and return of the index. We regress the returns of the index-tracking portfolio on returns of index without interception to get R square and Beta. Tracking error is the standard deviation of tracking difference. MSTD is mean square of tracking difference. In Year column, ‘all’ stands for years from 2006 to 2017.
Year Correlation R square Beta Tracking Error MSTD
Panel A: GSPC
2012 0.9797 0.9599 0.9620 1.94E-03 3.74E-06
In 2014, index-tracking portfolios of FSCSX and FBSOX have the lowest correlation and R square. Beta deviates from one much as well. Tracking error and MSTD are also relatively high. From the table 4.6, if we drop the return on ex-dividend date and the date before ex-dividend date, all measurements will dramatically improve. Hence one can see that paying dividends significantly affect the tracking measurements.
Table 4.6 Tracking Measurement (Drop Ex-Dividend Date)
Table 4.6 gives the tracking measurement of the index-tracking portfolio. The column, Drop Ex-Dividend Date, means dropping ex-dividend date the date before ex-dividend date in 2014. Correlation means the correlation between returns of index-tracking portfolio and return of the index. We regress the returns of the index-tracking portfolio on returns of index without interception to get R square and Beta. Tracking error is the standard deviation of tracking difference. MSTD is mean square of tracking difference.
Ticker
Drop Ex-Dividend
Date
Year Correlation R square Beta Tracking Error MSTD
Panel A: 22 Testing Period shows the ratio of tracking difference of index-tracking portfolio to tracking difference of the ETF. If the ratio is greater/less than one, the absolute tracking difference of
index-tracking portfolio is larger/smaller.4 For GSPC in panel A, index-tracking portfolio have less discrepancy in minimum in all years except 2009. And it tends to have large discrepancy in maximum. The similar situation occurs in IXIC index-tracking portfolio.
Table 4.8 and table 4.9 present the ratio of the specific tracking measurement of the index-tracking portfolio to the specific index-tracking measurement of the ETF. If the ratio is greater/less than one, the specific tracking measurement of index-tracking portfolio is larger/smaller. We convert Beta to absolute value of 1-Beta since we care about how Beta close to one. For GSPC in panel A, one thing that we would like to mention are that the ratio of Correlation and R square is less than one, and the ratio of Abs(1-Beta), Tracking Error and MSTD are less than one in all years with 66 testing period. It indicates that index-tracking portfolio tracks the index better when we consider all of the years.
However, we cannot make conclusion that the tracking ability of index-tracking portfolio is better or worse since the results fluctuate by year. For IXIC in panel B, no matter which testing period, the ratio of Correlation and R square is less than one, and the ratio of Abs(1-Beta), Tracking Error and MSTD are less than one in almost all years. It strongly indicates that the tracking ability of the index-tracking portfolio is better than the ETF.
Taking into account all these factors, we may reasonably come to the conclusion that the deep reinforcement learning model do a great job on index-tracking problem.
4 The tracking difference of the index-tracking portfolio and the tracking difference of the ETF have the same sign when they reach to maximum or minimum.
Table 4.7 Tracking Difference Ratio of Index-Tracking Portfolio and ETF Table 4.7 gives the ratio of tracking difference of the index-tracking portfolio to tracking difference of the ETF.
Year Min Max Min Max
Panel A:
GSPC 22 Testing Period 66 Testing Period
2006 0.23 0.39 0.23 0.31
Panel B: IXIC 22 Testing Period 66 Testing Period
2006 1.78 4.57 1.89 2.16
Table 4.8 Tracking Measurement Ratio of Index-Tracking Portfolio to ETF (Testing Period: 22 Trading Days)
Table 4.8 gives the ratio of a specific tracking measurement of the index-tracking portfolio with 22 testing period to a specific tracking measurement of the ETF.
Abs(1-Beta) is the absolute value of (1-Beta) which measures the beta how close to one.
Table 4.9 Tracking Measurement Ratio of Index-Tracking Portfolio to ETF (Testing Period: 66 Trading Days)
Table 4.9 gives the ratio of a specific tracking measurement of the index-tracking portfolio with 66 testing period to a specific tracking measurement of the ETF. Abs(1-Beta) is the absolute value of (1-Abs(1-Beta) which measures the beta how close to one.
Year Correlation R square Abs(1-Beta) Tracking Error MSTD
Panel A: GSPC