Overlay_dx: Automating Forecasting Evaluation

Traditional evaluation metrics for time series forecasting, such as MAE, RMSE, or MAPE, provide numerical values but often lack comprehensibility. These scalar summaries can hinder effective differentiation of model performances, especially when models have similar error magnitudes but different error distributions. The authors argue that a visual metric can complement numerical assessments by showing how predictions align with actual values over time and across confidence levels. They introduce **overlay_dx**, a metric that represents the percentage of predictions falling within a confidence interval around actual values. Once evaluation results are plotted, overlay_dx computes the area under the overlay curve, providing a quantitative measure of alignment between predicted and actual values across different thresholds and predictions. The goal is to offer a unified evaluation framework that combines visual and numerical assessments, enabling improved model comparison and insights for optimization in time series prediction.

The authors conducted extensive experiments to validate overlay_dx. They compared multiple time series prediction models, including statistical methods (e.g., ARIMA), machine learning models (e.g., gradient boosting), and deep learning architectures (e.g., LSTM, Transformer). The results show that overlay_dx and its AUC provide a more nuanced differentiation of model performance than traditional metrics. For instance, two models with similar RMSE can have significantly different overlay curves: one may consistently cover a high percentage of predictions at small thresholds, while the other may only achieve high coverage at larger thresholds. The visual overlay plot reveals patterns such as systematic bias (predictions consistently above or below actuals) and heteroscedastic errors (varying coverage across time). The AUC values correlate with but are not identical to traditional error metrics, offering complementary information. The authors report that overlay_dx is robust to outliers and scale, as it is based on relative coverage rather than absolute errors. They also demonstrate that the metric can be used for hyperparameter tuning: models optimized for overlay_dx AUC achieve bett