Ilmu Komputer & AI editorial
Open AccessOA2026
Evaluating Accuracy and Probabilistic Reliability of Zero-Shot Time Series Foundation Models
A benchmark study of six TSFMs reveals a fundamental trade-off between point accuracy and probabilistic calibration across energy, traffic, and financial datasets.
Panagiotis Michael; Moysis Symeonides; Demetris Trihinas· 2026· DOI 10.48550/arXiv.2609.25788
The core problem
Time Series Foundation Models (TSFMs) have emerged as a promising paradigm for zero-shot forecasting, eliminating the need for task-specific training. However, existing evaluations often focus solely on predictive accuracy, neglecting the critical aspect of probabilistic calibration. This study addresses this gap by benchmarking six TSFMs on energy, traffic, and financial datasets. The authors contrast TSFM performance against traditional statistical baselines and a supervised deep learning model. The central research question explores whether TSFMs can simultaneously achieve high point accuracy and reliable uncertainty quantification. The findings reveal a fundamental trade-off: while TSFMs generally outperform statistical and supervised models in accuracy, their probabilistic reliability varies significantly across architectures and forecasting horizons.
Innovation
Across all datasets, TSFMs outperform statistical baselines and the supervised DL model in point accuracy. However, probabilistic reliability diverges by architecture. xLSTM models exhibit robust calibration across all horizons, maintaining nominal coverage probabilities. In contrast, patch-based transformers achieve competitive accuracy but suffer from miscalibration at long horizons, often producing overconfident or underconfident prediction intervals. Transformer-based models show context saturation: beyond a certain context length, zero-shot performance plateaus or degrades. Quantitatively, xLSTM reduces calibration error by up to 30% compared to patch-based transformers at horizon 96. The trade-off is captured by the following relationship:
These results highlight that architectural choices critically influence the balance between generalization and uncertainty quantification.
Time Series Foundation Models (TSFMs) have emerged as a promising paradigm for zero-shot forecasting, eliminating the need for task-specific training. However, existing evaluations often focus solely on predictive accuracy, neglecting the critical aspect of probabilistic calibration. This study addresses this gap by benchmarking six TSFMs on energy, traffic, and financial datasets. The authors contrast TSFM performance against traditional statistical baselines and a supervised deep learning model. The central research question explores whether TSFMs can simultaneously achieve high point accuracy and reliable uncertainty quantification. The findings reveal a fundamental trade-off: while TSFMs generally outperform statistical and supervised models in accuracy, their probabilistic reliability varies significantly across architectures and forecasting horizons.
The study evaluates six TSFMs representing diverse architectural families, including xLSTM and patch-based transformers. Datasets span three domains: energy consumption, traffic flow, and financial time series. For each dataset, the authors generate zero-shot forecasts at multiple horizons. Performance is measured using standard point accuracy metrics (e.g., MAE, RMSE) and probabilistic calibration metrics (e.g., coverage probability, sharpness). Statistical baselines (e.g., ARIMA, exponential smoothing) and a supervised deep learning model (e.g., LSTM) serve as comparisons. The evaluation protocol ensures fair comparison by using identical train-test splits and horizon settings. The study also investigates context length effects, identifying saturation points where additional context does not improve zero-shot reasoning. The overall workflow is depicted below:
Why it matters
The observed trade-off between point accuracy and probabilistic reliability stems from architectural inductive biases. xLSTM's recurrent structure naturally models temporal dependencies and uncertainty, while patch-based transformers may overfit to local patterns, leading to poor long-horizon calibration. Context saturation in transformers suggests that zero-shot reasoning has limits; beyond a threshold, additional context introduces noise rather than signal. For real-world deployment, practitioners must prioritize either accuracy or reliability based on application requirements. For instance, risk-sensitive domains like finance may favor xLSTM for calibrated intervals, while short-horizon traffic prediction may benefit from patch-based transformers. The study recommends hybrid approaches and further research into calibration-aware training objectives. These findings provide evidence-based guidance for selecting TSFMs in practice.
Who should read this
CS practitioners and researchers
Opening member content…