Model WIS with a Tweedie family (#159) - #166
Open
kathsherratt wants to merge 3 commits into
Open
Conversation
The primary model used gaussian(link = "log"), which left deviance
residuals with skew 5.8 and kurtosis 77. Modelling log(WIS) directly
would fix the residuals but loses propriety of the score, so the fix had
to come from the error family instead.
Compare Gaussian, Gamma and Tweedie families on the joint specification,
holding formula and data fixed (R/sensitivity/check-family.R). Both
scales now use tw(link = "log"), replacing gaussian(log) on the log scale
and Gamma(log) on the natural scale.
On the log scale this is a large improvement: residual skew falls from
5.84 to 0.58, kurtosis from 77.5 to 9.2, deviance explained rises from
0.286 to 0.380. Gamma fits the same data almost identically (skew 0.52,
deviance explained 0.378) but does not converge on either scale, which is
the reason for preferring Tweedie. The power parameter is estimated at
1.99, the upper limit mgcv permits, so the fitted family is a Gamma in
all but numerical behaviour -- stated as such rather than implying p was
freely estimated.
This also resolves the natural-scale non-convergence recorded in the
previous commit: that was a Gamma problem, not a scale problem. On the
natural scale the change fixes convergence but not the fit -- residual
skew is unchanged at 4.59, because natural-scale WIS is skewed beyond
what any Tweedie can absorb. Nothing rendered reads output/natural/, so
no reported result is affected.
Several adjusted estimates moved materially, most notably the
deaths-versus-cases contrast (ratio 0.17 to 0.38). All substantive
conclusions hold: no model structure differs from the grand mean, stable
trends remain most predictable, increasing trends least, Omicron BA.1 the
hardest variant phase. Delta's interval now excludes 1.
Investigated whether the 1e-7 constant that process-data.R adds to every
score was driving the skew, since 553 forecasts (0.27%) score exactly
zero and the constant parks them 11 log-units below the next smallest
value. It was not: refitting with the constant removed and the exact
zeros retained changes residual skew by 0.01. process-data.R is therefore
unchanged and the result is recorded in the supplement as a negative
finding.
Also fixes:
- archive_diagnostics() reassigned its accumulator, so it returned the
whole fit-summary.csv rather than the row just written.
- tw() deparses its `link` argument, so passing a variable sent the
literal string "family_link"; use do.call to force the value through.
- Pre-existing typo in the results ("After adjustment forfeatures of the
forecast target ,").
The supplement comparison table carries a caption note that the Gamma fit
did not converge: it has the lowest residual skew of the three, and
without that note the table reads as though Gamma were the best choice.
Marks reviewer row #110 (QQ-plot dispersion) complete in the revision
tracker, which this work answers directly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The baseline-included-gaussian-log rows recorded the gaussian(log) and Gamma(log) fits from the previous commit. Both families are now covered by the family-* comparison arms, so the rows were redundant. Remove both rows, their archived appraise() panels, and update the CLAUDE.md pipeline example, which still named the retired label. The supplement comparison table filters on the family-* prefix, so it is unchanged and no re-render is needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Member
Having done the comparison is there a case for removing the 1e-7 offset for the "production" version? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compared Gaussian, Gamma and Tweedie families on the same joint specification (fixed formula and data).
Diagnostics
Using a Tweedie family created a large improvement. The Tweedie power parameter is estimated at 1.99 (the upper limit of
mgcv), so the fitted family is effectively Gamma.Comparison:
Also investigated whether the skew was a data artefact.
1e-7offset thatprocess-data.Radds to every scoreImpact
Both scales now use
tw(link = "log"), replacinggaussian(log)on the log scale andGamma(log)on the natural scale.Several adjusted estimates moved, particularly the deaths-versus-cases contrast (ratio from 0.17 to 0.38).
All substantive conclusions hold (no model structure differs from the grand mean).