Skip to main content

ABSTRACT

Quantifying uncertainty in seasonal forecasts is vital for informed decision-making.

Conformal prediction builds rigorous prediction intervals for any predictive model and any dataset.

• We compare its performance against a general parametric baseline.

• Conformal prediction demonstrates significantly better performance than the baseline on uncalibrated data and remains competitive on calibrated data, with more consistent interval widths observed.

Figure 1. Wind speed time series from observational reference, ensemble mean forecast and conformal prediction intervals on calibrated data.

METHODS

Data

Ensemble seasonal forecasts: 10 m wind speed forecasts with lead times from 1 to 6 months generated by ECMWF-SEAS5.

Observational reference: Vortex SERIES.

70 locations worldwide.

Training period from 1993 to 2016. Testing period from 2017 to 2024.

Calibration

We define two datasets:

Uncalibrated: original forecasts without calibration.

Calibrated: original forecasts calibrated using Ensemble Model Output Statistics (EMOS), which corrects the bias and underdispersion often shown by seasonal forecasts.

Prediction intervals

• We set a confidence level of 95%.

• We define two methods to build prediction intervals:

  • Baseline: 2.5th and 97.5th percentiles assuming the forecast distribution is Gaussian.
  • Conformal: result of applying Conformal Prediction. Ensemble Batch Prediction Intervals (EnbPI) is deployed, tailored specifically for dynamic time-series. It establishes prediction intervals by assessing errors between past forecasts and observations.

Evaluation metrics

• Coverage: percentage of times the observed value falls within the prediction interval (ideally 95%).

• Width: width of the prediction interval (smaller widths are preferable).

RESULTS

On uncalibrated data, conformal prediction significantly outperforms the baseline. This is attributed to the inherent bias and underdispersion in the forecast distribution, rendering the distribution’s percentiles unreliable.

On calibrated data, conformal prediction performs similarly to the baseline, yet showcases more consistent interval widths across locations, indicated by smaller error bars.

Figure 2. Coverage and width of prediction intervals for two datasets and two methods.

CONCLUSIONS

Conformal prediction demonstrates to be an effective method to build prediction intervals, whether the seasonal forecast is calibrated or not.

Overall, it is preferred over the parametric baseline because it does not make any assumption about the data and yields more consisted interval widths.

Do you want to know more?

"*" señala los campos obligatorios

Este campo es un campo de validación y debe quedar sin cambios.