Reinstate baseline mode; preserve alternative model fitting diagnostics - #164
Open
kathsherratt wants to merge 2 commits into
Open
Reinstate baseline mode; preserve alternative model fitting diagnostics#164kathsherratt wants to merge 2 commits into
kathsherratt wants to merge 2 commits into
Conversation
so baseline is included everywhere as a valid model
The baseline and the ensemble had been excluded together by a single
grepl("EuroCOVIDhub-") filter, but they are not equivalent. The ensemble
is a function of the contributed forecasts and would double-count them;
the baseline is an independently specified statistical model and belongs
in the sample on the same terms as any other participant.
Narrow the filter at every remaining site and relabel the flow-diagram
exclusion step from "Not created by Hub" to "Not the Hub ensemble".
The sample grows from 47 to 48 models across 38 teams, with the
statistical group going from 12 to 13. Because the baseline submitted for
every country in almost every week, that group's share of forecasts rises
to ~41%, so the previous claim that mechanistic, semi-mechanistic and
statistical models each contributed about a third no longer holds and is
replaced. Model counts, rater-disagreement counts and per-structure
forecast shares in the results text are now computed inline from the data
rather than hardcoded.
Add a spec_label argument to model_wis(). When supplied it archives the
appraise() panel plus a row of fit statistics (family, link, formula, n,
AIC, deviance explained, deviance-residual skew and kurtosis) under
output/diagnostics/, upserted on (spec_label, scale). The supplement
still reads the stable output/<scale>/plots/check_joint.png path. This
exists so the specifications planned next -- a skew-tolerant error family
(#159) and a method-by-target interaction (#158) -- can be compared
against this fit rather than silently overwriting it.
Also correct CLAUDE.md, which pointed at a supplement path that does not
exist, and gitignore the stray Rplots.pdf that non-interactive Rscript
runs leave behind.
Note: the natural-scale joint Gamma fit reports "algorithm did not
converge" from bgam.fitd. Effects are finite and structurally sane, and
nothing in the rendered manuscript or supplement reads output/natural/,
so this does not affect reported results -- but it needs resolving as
part of the error-family work in #159.
Note: R/sensitivity/check-autocorrelation.R sources model-logresp.R,
deleted in an earlier commit, so it does not currently run. Its filter is
narrowed for consistency but the script needs rebasing onto another
residual source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sbfnk
approved these changes
Jul 29, 2026
sbfnk
left a comment
sbfnk
left a comment
Member
There was a problem hiding this comment.
some minor text suggestions, as I thought it was a bit overindexed on the baseline model inclusion
Comment on lines
+12
to
+13
| We excluded the ensemble model created by the Hub team, because it is a function of the contributed forecasts and would therefore double-count them. | ||
| We retained the Hub baseline model, which is an independently specified statistical model and so contributes to the sample on the same terms as any other participant. |
Member
There was a problem hiding this comment.
Suggested change
| We excluded the ensemble model created by the Hub team, because it is a function of the contributed forecasts and would therefore double-count them. | |
| We retained the Hub baseline model, which is an independently specified statistical model and so contributes to the sample on the same terms as any other participant. | |
| We excluded the ensemble model created by the Hub team, because it is a function of the contributed forecasts and would therefore double-count them, but retained the Hub baseline model as an independently specified statistical model contributing on the same terms as any other participant. |
Comment on lines
+13
to
+14
| We included the Hub baseline model, which is a deliberately naive statistical model submitting for every country in almost every week. | ||
| It therefore contributes a large share of the statistical group's forecasts, and anchors that group's estimate towards the performance achievable without any epidemiological structure. |
Member
There was a problem hiding this comment.
I don't feel super strongly but I don't think this needs specific highlighting
Suggested change
| We included the Hub baseline model, which is a deliberately naive statistical model submitting for every country in almost every week. | |
| It therefore contributes a large share of the statistical group's forecasts, and anchors that group's estimate towards the performance achievable without any epidemiological structure. |
Comment on lines
+74
to
+75
| We also excluded the ensemble model created by the Hub team, because it is constructed from the contributed forecasts and so would double-count them. | ||
| We retained the Hub baseline model: it is an independently specified statistical model, and including it gives a naive reference point against which participant forecasts can be read. |
Member
There was a problem hiding this comment.
Suggested change
| We also excluded the ensemble model created by the Hub team, because it is constructed from the contributed forecasts and so would double-count them. | |
| We retained the Hub baseline model: it is an independently specified statistical model, and including it gives a naive reference point against which participant forecasts can be read. | |
| We also excluded the ensemble model created by the Hub team, because it is constructed from the contributed forecasts and so would double-count them, but retained the Hub baseline model as an independently specified statistical model. |
Member
|
Only other thing would be to check if anything in the supplement etc. needs to be updated / re-generated with the additional model |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In this PR: