VaR Backtesting
The VaR backtest screen runs two tests. In both cases, we are comparing the number of exceedances, versus the predicted number of exceedances (an exceedance is a day when the P&L exceeded the VaR, which is calculated prior to the start of the day, typically at the end of the previous day).
Test of Overall Frequency
The first test, “Test of Overall Frequency”, looks at the total number of exceedances in the period, and outputs the probability of seeing that many exceedances. More specifically it outputs the probability of seeing that many exceedances or more, or that many exceedance or less.
For example, if we are measuring 1-day 95% VaR, we would expect to see 5 exceedances over 100 days, but we could easily see 4 or 6. The probability of seeing 4 or less is 43.60%, while the probability of seeing 4 or more is 74.22% (note the two probabilities add up to more than 100% because both include the probability of observing exactly 4 exceedances). On the other hand, the probability of seeing 0 or 12 exceedances is very unlikely. The probability of seeing 12 or more exceedances is just 0.43%.
The application reports the number of data points (“# of points”) in the lookback window, the number of exceedances observed over that time (“# exceedances”), and the number of exceedances expressed as a percentage (“% exceedances”; if you are testing 95% VaR, this number should be close to 5%). The application then reports the probability of seeing the observed number of exceedances or more (“P[X >= p]”) and the probability of seeing that many or less (“P[X <= p]”).
At the fund level, if either of the probabilities (“P[X >= p” or “P[X <= p]”) is less than 5%, the result should be investigated further. This could be a sign of bad data or an inappropriate model being used. Too much trading activity can cause more or less exceedances than expected. The backtesting assumes that there is no trading during the day. Also, if you look at lots of positions or sub-portfolios, you are likely to see some that have too few or too many exceedances, but you should not see too many.
Test of Conditional Independence
The second test, “Test of Conditional Independence”, tests the probability of seeing two exceedances in a row. If we are testing 1-day 95% VaR, then over 100 days we expect to see 5 exceedances. It is rarely going to be the case that the exceedances are perfectly spaced, occurring exactly once every 20 days. On the other hand, it would be strange if we had 5 exceedances in a row followed by 95 days with no exceedances. This bunching together of exceedance events was a problem that plagued early VaR models.
Technically, we want exceedance events to be independent of each other. If the 95% VaR is $678, then this should mean that there is a 5% probability of losing $678 or more tomorrow. It shouldn’t matter that there was an exceedance the day before or not.
Just as we tested the overall number of exceedances, one way that we can test for independence is to look at how often we see two exceedances in a row. If we are testing the 95% VaR over 2,000 days we would expect to see 100 exceedances. If we did see exactly 100 exceedance, we would expect to see 5 of those occurring immediately after another exceedance.
Just as with the overall test, the application reports the probability of seeing more or less than the observed number of events. In this case, though it is the probability of seeing the number of next-day exceedances.
One problem with this test is that there are not likely to be a lot of data points to work with. As we saw in our example, for 95% VaR, we need 2,000 days of observations, approximately 8 years, just to observe 5 next-day exceedance events.
Further Reading
For more on VaR backtesting, see our white paper, An Introduction to Value at Risk.