Validating three methods for making probability statements in fisheries forecasts.

We report on a test of three methods that have been used by ICES in making probability statements about fisheries forecasts to management agencies (XSA/WGMTERM, ICA/ICP, and a stochastic projection method that was first applied to North Sea herring). The principle of the test is to calculate many stock assessments and forecasts using historic data, and to compare probability statements about some quantities (forecast biomass, relative biomass, catch etc.) with the latest and presumably more reliable estimates. For most ICES and NAFO stocks where a sufficiently long time-series exists, we calculate retrospective (8 year before present) assessments and then calculate medium term forecasts 5 years forwards. By comparing the frequencies of expected and actual outcomes, we conclude that some methods can be used to make reasonably reliable probability statements about relative biomass (e.g. biomass relative to biomass at some time in the past), but that for most other quantities and most methods the accuracy of probability estimates is very poor. In general, the methods tended to underestimate uncertainty and there was a relatively large proportion of "unexpected" outcomes in the forecasts