Understanding Performance Metrics

2025-10-26 • NHLForecasts

The Performance page contains several kinds of evaluation. Read the label and sample size before comparing any numbers.

Prospective Records and Historical Diagnostics

The prospective record uses timestamped forecasts saved before scheduled puck drop. It starts with new captures; older predictions are not backfilled into it. Each game contributes one qualifying forecast, and model versions are reported separately.

Historical summaries may include training games or regenerated predictions. They help diagnose model behavior but do not establish how it performed on unseen games. Cross-validation and xG holdout results are separate evaluations with their own samples and timing limitations.

Daily, 30-day, and season windows answer different questions. Short windows move sharply with a few outcomes. Longer windows contain more observations, but can mix changing team strength and model behavior.

Accuracy and Baselines

Accuracy is the fraction of games in which the predicted winner won. Independent random winner selection has an expected accuracy of 50%. Always selecting the home team has an accuracy equal to the home win rate in the evaluated sample.

Compare models on the same games. A higher accuracy alone does not establish a meaningful improvement; sample size, uncertainty, and how the forecasts were collected matter.

The average probability assigned to the eventual winner describes confidence after observing results. It does not establish calibration and should be read alongside Brier score, log loss, and calibration tables.

Brier Score

For a home win probability p and outcome y (1 for a home win, 0 otherwise), the binary Brier score is the average of (p − y)². Lower is better. Forecasting 50% for every game gives a score of 0.25.

If the home team wins, a 51% forecast has squared error 0.2401 and a 95% forecast has squared error 0.0025. If the home team loses, that 95% forecast has squared error 0.9025. Accuracy counts each correct winner equally; Brier score also evaluates the probability assigned.

Brier score reflects more than calibration. A low score by itself does not prove that each probability range is reliable.

Log Loss

Log loss is the average negative natural logarithm of the probability assigned to the observed outcome. Lower is better. A 50% forecast for every game gives about 0.6931. Assigning a very small probability to an outcome that occurs produces a large penalty.

Total Goals

Total-goals probabilities can be evaluated against an event such as six or more goals. Grouping those probabilities lets us compare mean forecasts with the fraction of games in which the event occurred.

MAE is the average absolute difference between a predicted total and the result. RMSE is the square root of the average squared difference, so larger misses receive more weight. Both are expressed in goals and lower is better. An RMSE of two goals does not imply that any fixed proportion of forecasts falls within two goals of the result.

Calibration and Uncertainty

Calibration asks whether events assigned similar probabilities occur at roughly those rates. If 100 forecasts all give the home team a 60% chance, 60 home wins would match that group's forecast exactly. It would still be a sample estimate, not proof that future 60% forecasts will behave identically.

Two models can both pick 60% of winners correctly while assigning every predicted winner either 60% or 90%. The first matches that group's observed win rate; the second is overconfident in that sample.

The prospective report shows sample sizes and approximate 95% Wilson intervals for accuracy and calibration-bin win rates. These intervals assume independent games. Repeated teams, shared dates, and changing conditions can add uncertainty they do not capture. They are not uncertainty intervals around an individual game's forecast.

There is no universal three-point or ten-point cutoff separating noise from bias. Looking across many teams and venues also increases the chance of finding a large gap by chance. Treat those gaps as questions for further evaluation, not automatic probability adjustments.

Other Evaluation Tables

Starter, team, venue, and in-game checkpoint tables describe different subsets of the data. Differences can reflect the teams, dates, and situations in those subsets; they do not by themselves identify a cause.

For xG, shot-level holdout metrics measure a different prediction task from game winners. A chronological split helps separate earlier training examples from later evaluation examples, but input availability and data revisions still need checking. An aggregate goals-to-xG ratio close to one does not establish calibration across shot types or probability ranges.

Playoff results are reported separately because they cover a smaller and different set of games. Read them with their sample size and evaluation method in view.

Use the prospective record to follow forecasts as results arrive. Use historical diagnostics to investigate patterns, with the evaluation limits attached.

← Back to Articles