Validating the backtests of risk measures

Paper presented at INFINITI Conference on International Finance 11-12 June 2007, Trinity College Dublin. Also presented at Financial Management Association Annual Meeting, Orlando, Florida , 18-20 October 2007Financial risk model evaluation or backtesting is a key part of the internal model’s approach to market risk management as laid out by the Basle Committee on Banking Supervision (2004). However there are a number of backtests that may be applied and there is little guidance as to the most appropriate method. The goal of this paper is to analyze the ability of various evaluation methodologies to gauge the accuracy of risk models. We compare evaluation effectiveness using the standard binomial approach, together with the interval forecast backtesting, the density forecast backtesting and the probability forecast backtesting. Our comparison is completed for three risk measures: Value-at-Risk (VaR), Expected Shortfall (ES) and Spectral Risk measure (SRM). We pay special attention to applications related to ES and SRM as backtesting of these models have not been explored in any detail thus far. Based on the Monte Carlo simulations and the empirical study, a number of interesting results emerge. Firstly within hypothesisbased tests, including the binomial backtesting, the interval forecast backtesting and the density forecasts backtesting, the overall dominance of density forecast backtesting is confirmed. In particular, the backtesting for SRM and ES is more effective than for VaR in identifying an incorrect model from alternative models in a small sample setting. Secondly, we propose a loss function for SRM where the probability forecast backtesting is capable of identifying accurate models from alternative models. Thirdly, in all of the backtesting methods examined, the choice of the distribution specification is a more important factor in determining the evaluation performance than the choice of the volatility specification.University College Dublin. School of Busines