Frozen v1.1 against challenger tl-v2-candidate-1
Does any new idea beat the live model?
Every idea is a hypothesis until it improves forecasts on matches it was not tuned on. The rules were committed before any result (protocol). These are historical simulations, never part of the live record.
Ideas tested
Tuning seasons 2013-14, 2014-15, 2015-16, 2016-17
| Idea | Change in log loss | Interval (Bonferroni) | Result |
|---|---|---|---|
| Low-score dependence (Dixon–Coles ρ) | +0.0003 | -0.0013 to 0.0019 | rejected |
| Promoted-club prior | −0.0023 | -0.0075 to 0.0029 | rejected |
| Championship-informed promoted prior | +0.0042 | 0.0008 to 0.0079 | rejected |
| Club-specific home advantage | −0.0000 | -0.0009 to 0.0007 | rejected |
| Short rest across competitions (EPL + UCL schedule; PARTIAL coverage) | +0.0000 | -0.0005 to 0.0007 | rejected |
| Negative control: kickoff slot | +0.0009 | -0.0018 to 0.0036 | rejected |
| Rating half-life | −0.0010 | -0.0032 to 0.0012 | rejected |
An idea is accepted only if the whole interval is below zero (better forecasts). The kickoff-slot idea is a deliberate negative control.
Frozen v1.1 against the candidate
Holdout A (out-of-period, untouched)
Neither model was tuned on these seasons. Scored once, after the candidate was frozen. 2017-18, 2018-19, 2019-20: 1140 matches on 328 match days.
| Metric | Frozen v1.1 | Candidate v2 | v2 − v1.1 (95% interval) | Base rates |
|---|---|---|---|---|
| Three-way log loss (lower is better) | 0.9444 | 0.9444 | +0.0000 (-0.0000 to 0.0000) | 1.0593 |
| Brier score, three-way | 0.5578 | 0.5578 | +0.0000 (-0.0000 to 0.0000) | 0.6399 |
| Ranked probability score | 0.1925 | 0.1925 | +0.0000 (0.0000 to 0.0000) | 0.2313 |
| Calibration slope (1 is ideal) | 1.16 | 1.16 | ||
| Super Tips (frozen policy) | 662/932 (71%, expected 69%) | 662/932 (71%, expected 69%) | ||
| Pass rate | 18% | 18% | ||
| Market (closing odds) benchmark | not measured: no lawful archived odds present (see docs/v2/SOURCES.md) |
Coverage: rest schedule complete for 0% of matches (no Europa League or cup data); 324 matches involved a promoted club (108 with Championship data). Intervals: paired bootstrap over match days.
Brier skill by market
| Market | v1.1 | v2 |
|---|---|---|
| 1X2:HOME | 16.2% | 16.2% |
| 1X2:DRAW | 2.5% | 2.5% |
| 1X2:AWAY | 17.4% | 17.4% |
| DC:1X | 17.4% | 17.4% |
| DC:X2 | 16.2% | 16.2% |
| OU:OVER:2.5 | 1.9% | 1.9% |
| OU:UNDER:2.5 | 1.9% | 1.9% |
| OU:UNDER:3.5 | 1.9% | 1.9% |
| OU:OVER:1.5 | 1.2% | 1.2% |
| BTTS:YES | 0.2% | 0.2% |
| BTTS:NO | 0.2% | 0.2% |
| TEAM:HOME:OVER:1.5 | 11.5% | 11.5% |
| TEAM:AWAY:OVER:1.5 | 10.6% | 10.6% |
| TEAM:HOME:UNDER:0.5 | 5.4% | 5.4% |
| TEAM:AWAY:UNDER:0.5 | 7.1% | 7.1% |
Holdout B (in-period, reused)
Not a clean holdout: v1.1 was tuned on 2022-24 and its rule chosen after seeing 2024-27. Reported for completeness. 2021-22, 2022-23, 2023-24, 2024-25, 2025-26, 2026-27: 1950 matches on 600 match days.
| Metric | Frozen v1.1 | Candidate v2 | v2 − v1.1 (95% interval) | Base rates |
|---|---|---|---|---|
| Three-way log loss (lower is better) | 0.9809 | 0.9809 | +0.0000 (-0.0000 to 0.0000) | 1.0710 |
| Brier score, three-way | 0.5836 | 0.5836 | +0.0000 (-0.0000 to 0.0000) | 0.6482 |
| Ranked probability score | 0.2011 | 0.2011 | +0.0000 (0.0000 to 0.0000) | 0.2326 |
| Calibration slope (1 is ideal) | 1.10 | 1.10 | ||
| Super Tips (frozen policy) | 1145/1638 (70%, expected 69%) | 1144/1638 (70%, expected 69%) | ||
| Pass rate | 16% | 16% | ||
| Market (closing odds) benchmark | not measured: no lawful archived odds present (see docs/v2/SOURCES.md) |
Coverage: rest schedule complete for 0% of matches (no Europa League or cup data); 554 matches involved a promoted club (554 with Championship data). Intervals: paired bootstrap over match days.
Brier skill by market
| Market | v1.1 | v2 |
|---|---|---|
| 1X2:HOME | 13.9% | 13.9% |
| 1X2:DRAW | 0.9% | 0.9% |
| 1X2:AWAY | 13.1% | 13.1% |
| DC:1X | 13.1% | 13.1% |
| DC:X2 | 13.9% | 13.9% |
| OU:OVER:2.5 | 1.3% | 1.3% |
| OU:UNDER:2.5 | 1.3% | 1.3% |
| OU:UNDER:3.5 | 1.2% | 1.2% |
| OU:OVER:1.5 | 0.2% | 0.2% |
| BTTS:YES | 0.7% | 0.7% |
| BTTS:NO | 0.7% | 0.7% |
| TEAM:HOME:OVER:1.5 | 9.4% | 9.4% |
| TEAM:AWAY:OVER:1.5 | 6.0% | 6.0% |
| TEAM:HOME:UNDER:0.5 | 7.3% | 7.3% |
| TEAM:AWAY:UNDER:0.5 | 4.2% | 4.2% |
v1.1 calibration on untouched seasons
Promotion criteria
- Not met: Holdout A log loss better, 95% interval excluding zero
- Not met: Holdout A Brier and RPS not worse; calibration slope 0.9–1.1
- Met: Holdout B not worse beyond noise (upper 95% bound of v2 − v1.1 below +0.002)
- Not met: Forward test: ≥ 8 rounds of shadow forecasts, not worse (not started: shadow forecasting begins with Matchweek 10)
- Not met: Owner approval, new model version, freeze-log entry (not requested)
Not tested, and why
- Expected goals / shot quality: no lawful, documented, archived source reachable (docs/v2/SOURCES.md).
- Team news and player impact: no player-level availability history; absences are shown as facts with zero effect.
- Weather: no archived as-of forecasts available here; context only by design.
- Betting odds as a benchmark: no archived closing odds present; market comparison not measured.
- Head-to-head records: excluded by design: small samples, explained by team strength.
- Europa League, Conference League and domestic cup congestion: no open historical data; rest is PARTIAL.
Generated 2026-10-31 10:32 UTC by npm run v2:evaluate. How the live model works.