CHAPTER 4:RESEARCH METHOD
4.4 Back Testing
4.4.2 Christoffersen
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
4.4.2 Christoffersen
Christoffersen (1998) extends the LRuc statistic to specify that the deviation must be serially independent. The test is set up as following:
Deviation indicator= 0 if VaR is not exceeded;
Deviation indicator= 1 otherwise.
Then we define Tij as the number of days in which state j occurred in one day while it was at I the previous day andπi as the probability of observing an exception conditional on state i the previous day. Table 5 shows how to construct a table of conditional exceptions.
If today’s occurrence of an exception is independent of what happened the previous day, the entries in the second and third columns should be identical. The relevant test statistic is
[ ] [ ]
Here, the first term represents the maximized likelihood under the hypothesis that exceptions are independent across days, or
The second term is the maximized likelihood for the observed data.
Table 5. Building an Exception Table: Expected Number of Exceptions Conditional
Day Before
No Exception Exception Current day
No exception
Exception )
Total
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
The combined test statistic for conditional coverage then is
LRcc = LRuc + LRind
Each component is independently distributed as x2(1) asymptotically. The sum is distributed as x2(2). Thus we should reject at the 95 precents test confidence level if LR>5.991. We would reject independence alone if LRind>3.84.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y Chapter 5: Results
The time series models are estimated each day with data available up to that point. To obtain stable estimates for the model, forecasts for 2011 (days 1 through 243) are in-sample. Rolling out-of-sample forecasts starts after 2012. Out-of-sample estimates are updated daily.
Here, we adopt two kinds of reduced-form models; they are fitted model and Berkowitz & O’Brien model (BO Model). The fitted model uses the first 243-day data as the in-sample to fit a time series model; and the BO model follows Berkowitz
& O’Brien (2002) using a ARMA(1,1)-GARCH(1,1) as the time series model.
Given parameters estimates, we forecast the next day’s 95% and 99% VaR.
The results of the forecast, both within and out-of-sample, are shown in Figure 5 by the grey line, along with P&L by the dotted line and internal model by solid line. As we can see one-day ahead reduced-form forecasts appear to track the lower tails of P&L really well compared to the internal structural model. It tracked the huge P&L drop in 2011, which did not caught by the internal model. This shows that time series model does better at adjusting in volatility through time.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
95% VaR of Fitted Model
95% VaR of BO Model
Figure 5. P&L, 95% Internal VaR, Fitted VaR, and BO VaR Note: Data are expressed in standard deviations.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
99% VaR of Fitted Model
99% VaR of BO Model
Figure 6. P&L, 99% Internal VaR, Fitted VaR, and BO VaR Note: Data are expressed in standard deviations.
‧
Summary statistics and backtests for three models are presented in Table 6 and Table 7 as following.
The third column of Table 6 shows that the time series models remove first-order persistence successfully. Table 6 also shows that both time series models can have lower mean VaR and max VaR in 99th percentile VaR. The phenomenon might be interpreted that time series models have better performance in the fat-tailed situation. But, in 95th percentile VaR, that phenomenon disappears.
The higher mean VaRs mean the internal models are more conservative and should generate lower mean violation and max violation. However, the mean violation and max violation, shown in column 8 and 9, exhibit that is not the case.
Even though the internal model is more conservative, the time series models still can have lower mean and max violation.
This result indicates a potentially important advantage for the reduced-form model. Since the magnitudes of the VaR forecasts are used for determined economic capital for the insurance companies. The reduced-form time series models are able to deliver lower required capital requirement without having large violations. This reflects the reduced-form models have greater responsiveness to the P&L volatility.
Table 6. Summary Statistics of Three Models
Summary
Note: Box-Ljung statistics are for first-order serial correlation. The Internal Model are calibrated by insurance company. In column 2, the fitted model is ARMA(1,1)-ARCH(1); in column 3, the model is ARMA(1,1)-GARCH(1,1). The grey shading parts have better performance compared to Internal Models.
‧
results provide little basis to distinguish between the time series models and internal model.For 95th percentile VaR, all methods are rejected in terms of coverage. And, for the independence, only the 95th percentile VaR of internal model is rejected. The main reason of that is because there are 4 continuous violations coming after previos –day violation in this period. And, for 99th percentile VaRs of all methods are not rejected.
Table 7. Backtests for Three Models
Backtests Violation
Note: P-values are demonstrated in square brackets. The 5% critical value is 3.84, and the 1% critical value is 6.64. * and ** stand for significance at the 5 and 1 percent levels, respectively.
To have a further understanding of this reduced-form model, this paper separates the data into three parts, 2011, 2012, and 2013. And the results of these models are listed in Table 8 to Table 13 as following.
We follow the same fashion with the previous method. Using fitted model and B.O. model as reduced-form time series models. For both models, we use the first half data as in-sample and the rest as out-of-sample. Take 2011’s data as an example, we have 243 P&L data, so we use the first 120 data to fit time series models for fitted model and B.O. model.
‧
Even though they have lower max violations, they have greater mean violations too.
That means the fitted models do not outperform the internal models. In addition, the backtests in Table 9 also shows that the 95th percentile VaR of fitted model is the only model rejected in the coverage test in 2011.
And for the B.O. model in 2011, it has the same result as in all samples. 99th percentile VaR has a better performance compared to the internal model, and it passes both coverage and independence tests in backtests.
Table 8. Summary Statistics of Three Models in 2011
Summary
Note: Box-Ljung statistics are for first-order serial correlation. The Internal Models are calibrated by insurance company. In column 2, the Fitted Model is ARCH(1); in column 3, B.O. Model is ARMA(1,1)-GARCH(1,1). The grey shading parts have better performance compared to Internal Model.
‧
Table 9. Backtests of Three Models in 2011
Backtests Violation Rate Coverage Conditional
Coverage
Note: P-values are demonstrated in square brackets. The 5% critical value is 3.84, and the 1% critical value is 6.64. * and ** stand for significance at the 5 and 1 percent levels, respectively.
And in 2012, the results in summary statistics have a different fashion. In Table 10, the fitted models still barely have violations. This time, compared to internal models, fitted model and B.O. model have better performance in the 95th percentile VaRs in terms of mean VaR, mean violation, and max violation. This fashion is totally different form the total sample and sample in 2011- 99th percentile VaRs of most models have better performances. Moreover, in Table 11, B.O. model passes the coverage test in 95th percentile VaR in backtest.
‧
Table 10. Summary Statistics of Three Models in 2012
Summary
Note: Box-Ljung statistics are for first-order serial correlation. The Internal Models are calibrated by insurance company. In column 2, fitted model is ARMA(3,3); in column 3, B.O. Model is ARMA(1,1)-GARCH(1,1). The grey shading parts have better performance compared to Internal Models.
Table 11. Backtests of Three Models in 2012
Backtests Violation Rate Coverage Conditional
Coverage Independence
Note: P-values are demonstrated in square brackets. The 5% critical value is 3.84, and the 1% critical value is 6.64. * and ** stand for significance at the 5 and 1 percent levels, respectively.
In the last year, expressed in Table 12, the summary statistics of all reduced-form models do not outperreduced-form the internal models in either percentile.
‧
violations in 99th percentile VaRs compared to the previous tests. As the result, both of them cannot pass the coverage tests.In Table 13, we can see the violation rates of fitted model and the BO model in 99th percentile VaRs are obliviously over 1%.
Table 12. Summary Statistics of Three Models in 2013
Summary
Note: Box-Ljung statistics are for first-order serial correlation. The Internal Models are calibrated by insurance company. In column 2, the Fitted Model is ARCH(1); in column 3, the B.O. Model is ARMA(1,1)-GARCH(1,1).
Table 13. Backtests of Three Models in 2013
Backtests Violation Rate Coverage Conditional
Coverage Independence
Note: P-values are demonstrated in square brackets. The 5% critical value is 3.84, and the 1% critical value is 6.64. * and ** stand for significance at the 5 and 1 percent levels, respectively.
‧
In the last, we try different ways to compute VaR for time series method. First, we try moving window method (M.W. method). In this method, we use the first 243 samples as the in-sample to fit the time series model, and we forecast the first half year of 2012. Following that, we move the data window to 120-367 as in-sample to fit the second time series model, and we forecast the second half year of 2012.
Second, we use the data in 2011 as the in-sample to fit a time series model and forecast the VaR in 2012 (2011 Forecast). Last, we use the 2011 data as the in-sample to fit the Berkowitz and O’Brien’s ARMA(1,1)-GARCH(1,1) model and forecast the VaR in 2012 (2011 B.O. Forecast). The results are demonstrated in the Table 14 and 15.
In Table 14, we can see that the MW method can use a lower mean VaR to generate lower mean violation compared to internal model in 99th percentile VaR.
And the mean violation is the lowest in all method we have tried in this paper. In addition, it passes both backtests. However, it cannot generate a lower max violation at the same time. For 2011 forecast and 2011 B.O. forecast models, both of them do not have violation in 99th percentile VaR and do not outperform in 95th percentile VaR.
Table 14. Summary Statistics for Internal Model and Other Methods
Summary
Note: The Internal Models are calibrated by insurance company. The first and the second models are both ARMA(1,1)-ARCH(1) in M.W. method. The model of 2011 forecast is ARMA(1,1)-ARCH(1).
The B.O. 2011 forecast model is ARMA(1,1)-GARCH(1,1).
‧
Table 15. Backtests of Internal Model and Moving Window Method in 2012
Backtests Violation Rate Coverage Conditional
Coverage Independence
Note: P-values are demonstrated in square brackets. The 5% critical value is 3.84, and the 1% critical value is 6.64. * and ** stand for significance at the 5 and 1 percent levels, respectively.
‧
life insurance company in Taiwan. VaR is a really important equipment to quantify the risk for financial institutions nowadays. For insurance companies, they can use VaR method to generate the economic capital they need as capital buffer to survive the crisis.The structural model, adopted by most financial institutions, might have some limitations when computing VaR. Berkowitz and O’Brien (2002) found the models used by banks tend to be conservative. However, the losses can substantially exceed the VaR and such events tend to be clustered.
Moreover, the total amount of capital invested by insurance companies has grown substantially in the past few years. The market risk factors used by internal structural models, therefore, become larger and more complex. It is almost impossible for institutions to compute daily VaR considering joint distribution conditional on the current information.
This study is the first article that uses the univariate method with historical data of one life insurance company in Taiwan and provides its performance compared to the internal models that is actually in use. This paper follows the method of Berkowitz and O’Brien (2002) adopting a reduced-form model- time series model. It considers life insurance company’s trading P&L as one investment portfolio and reduces its risk factors to a univariate time series. The aim is trying to solve the aggregation problem of the internal structural model.
The followings are the general conclusions we have found in this paper,
1. On average, the time series models achieve the target violation rate in 99th percentile VaR coverage.
At the same time, the mean violations for the time series models are lower than the internal models. This result demonstrates the reduced-form time series models generally have better performances compared to the internal models. That can be interpreted as time series models track lower bound of P&L better than the internal models, since they do better at
‧
results when we separate the data in to annual data.2. Almost all 95th percentile VaR in every method cannot pass the coverage test in backtesting.
This result shows that the models we are using right now are too conservative. We can view this phenomenon in two perspectives. First, in the institutional perspective, the financial institutions cannot use their capital efficiently since the required capital they have to withdraw is higher. Second, in the supervisory perspective, the supervisors will not take this issue too seriously, since the financial institutions view the required capital in a conservative way. That means financial institutions will have more capital buffer to endure the financial crises.
3. The ARMA(1,1)-GARCH(1,1) model suggested by Berkowitz and O’Brien has the best performance in all the samples.
The reason is probably because this model considers both ARMA and GARCH model all the time, and this model fits the P&L of life insurance company really well. On the contrary, the fitted models sometimes will only have ARMA or only ARCH process in the model and have inferior forecasting performances. Hence, instead of fitting the time series model, the insurance companies can directly try ARMA(1,1)-GARCH(1,1) model to fit their P&L data.
4. The financial institutions can use the reduced-form model- time series model as a complementary method to the internal model while computing VaR for the following reasons.
First, time series models have a better performance tracking the lower bound of P&L. Compared to the reduced-form models, the internal VaRs did not adequately reflect changes in the P&L volatility. These results may reflect substantial computational difficulties in constructing large-scale structural models of market risks for large, complex portfolios.
Even the structural models permit firms to examine the effects of individual positions on market risk. Time series models may have
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
advantage in forecasting and as equipment for identifying the shortcoming of the structural models.
Second, the time series models are really easy to compute. This timesaving specialty can do a really good job as a complement to the structural models.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y Chapter 7: Suggestions
This paper suggests that if anyone wants to have a further research on this topic, he can amend the following points,
7.1 MORE OBSERVATIONS
In this paper, we only have one life insurance company’s around three years data. The future research can collect more observations in terms of depth and width.
In depth, the researcher can use more observations to come up with more robust results. In width, the researcher can collect data from more than one company or different types of companies, e.g. property insurance company.
7.2 CRISIS TEST
In this paper, our sample period, from Jan 1, 2011 to Feb 27, 2014, does not cover the 2008 financial crisis. If the future researcher included the crisis data, we can see the performance of the reduced-form model in the severe situation. Since VaR model aims to forecast the worst situation encountered by company, the crisis data will be a huge plus for the research.
7.3 VAR’S DRAWBACK
Since VaR is not a coherent risk measure (Artzner, 1999) and can lead to inconsistent results when aggregating capital. In order to understand the economic capital financial institutions need, we can test other risk measures, like Conditional Tail Expectation, a.k.a., CTE.
7.4 DIFFERENT FORECASTING METHOD
We have tried a lot of models and different two percentile VaRs to verify the performance of the reduced-form model compared to the internal models.
Nevertheless, there are still other forecasting methods have not been tested, like aggregated method, which aggregate all the past data as in-sample to fit the time series model.
‧
risk. Mathematical finance, 9(3), 203-228.Berkowitz, J., & O’Brien, J. (2002). How Accurate Are Value‐ at‐ Risk Models at Commercial Banks? The Journal of Finance, 57(3), 1093-1111.
Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity.
Journal of econometrics, 31(3), 307-327.
Christoffersen, P. F. (1998). Evaluating interval forecasts. International economic review, 841-862.
Christoffersen, P. F., & Diebold, F. X. (2000). How relevant is volatility forecasting for financial risk management? Review of Economics and Statistics, 82(1), 12-22.
Duffie, D., & Pan, J. (1997). An overview of value at risk. The Journal of derivatives, 4(3), 7-49.
Enders, W. (2008). Applied econometric time series. John Wiley & Sons.
Engle, R. F., & Manganelli, S. (2004). CAViaR: Conditional autoregressive value at risk by regression quantiles. Journal of Business & Economic Statistics, 22(4), 367-381.
Hendricks, D. (1996). Evaluation of value-at-risk models using historical data.
Federal Reserve Bank of New York Economic Policy Review, 2(1), 39-69.
Holton, G. A. (2002). History of Value-at-Risk: Working paper. Contingency Analysis, Boston.
Ian Farr, H. M., Mark Scanlon, Simon Stronkhorst. (February 2008). Economic Capital for Life Insurance Companies: Towers Perrin.
Jorion, P. (1997). Value at risk: the new benchmark for controlling market risk (Vol.
2): McGraw-Hill New York.
Jorion, P. (2002). How informative are value-at-risk disclosures? The Accounting Review, 77(4), 911-931.
Kupiec, P. H. (1995). Techniques for verifying the accuracy of risk measurement models. THE J. OF DERIVATIVES, 3(2).
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Lopez, J. A., & Walter, C. A. (2000). Evaluating covariance matrix forecasts in a value-at-risk framework.
Marshall, C., & Siegel, M. (1997). Value at risk: Implementing a risk measurement standard. The Journal of derivatives, 4(3), 91-111.
Zangari, P. (1997). Streamlining the market risk measurement process. RiskMetrics Monitor, 1, 29-35.
Chinese Literature
楊奕農, & 經濟. (2009). 時間序列分析: 經濟與財務上之應用. 雙葉書廊.