Probabilistic Contingency Tables: An Improvement to Verify Probability Forecasts

Sarah A. Gold
Edward D. White, Air Force Institute of Technology
William P. Roeder
Michael McAleenan
Christine M. Schubert Kabban, Air Force Institute of Technology
Darryl K. Ahner, Air Force Institute of Technology

The "Link to Full Text" on this page will load the full-text article at the AMS website. A PDF of the article is available from that site.

Abstract

The 45th Weather Squadron (45 WS) records daily rain and lightning probabilistic forecasts and the associated binary event outcomes. Subsequently, they evaluate forecast performance and determine necessary adjustments with an established verification process. For deterministic outcomes, weather forecast analysis typically utilizes a traditional contingency table (TCT) for verification; however, the 45 WS uses an alternative tool, the probabilistic contingency table (PCT). Using the TCT for verification requires a threshold, typically at 50%, to dichotomize probabilistic forecasts. The PCT maintains the valuable information in probabilities and verifies the true forecasts being reported. Simulated forecasts and outcomes as well as 2015–18 45 WS data are utilized to compare forecast performance metrics produced from the TCT and PCT to determine which verification tool better reflects the quality of forecasts. Comparisons of frequency bias and other statistical metrics computed from both dichotomized and continuous forecasts reveal misrepresentative performance metrics from the TCT as well as a loss of information necessary for verification. PCT bias better reflects forecast verification in contrast to that of TCT bias, which suggests suboptimal forecasts when in fact the forecasts are accurate.