Forecast skill requires the same cohort baseline¶
GA-SYN-012 · Edition 2026.10.9 · Synthetic technical investigation
Proposition and operational question¶
For the specified four outcomes, constant quarter-probability forecasts have Brier loss five-sixteenths and skill minus one-quarter relative to the same-cohort half-probability baseline.
This matters when a continuing task reuses an earlier decision or representation. A final success label can conceal a different failure earlier in the trace. The proposition therefore identifies the exact observation that would refute it.
Definitions, trust and failure model¶
Outcomes are zero, zero, one and one. All forecasts equal one quarter. Brier loss is the mean squared difference between forecast and outcome. Skill is one minus forecast loss divided by baseline loss. The baseline is the empirical half-probability forecast on this same cohort. Exact rational arithmetic avoids rounding ambiguity. No population-level inference is made.
Worked synthetic trace¶
The two zero outcomes each contribute one-sixteenth; the two one outcomes each contribute nine-sixteenths. The mean is twenty-sixty-fourths, or five-sixteenths. The baseline contributes one-quarter on each outcome. Dividing five-sixteenths by one-quarter gives five-fourths; subtracting from one gives minus one-quarter. The forecasts underperform this declared baseline.
Negative control¶
The mutant uses baseline loss one-half from a different comparison. The reported skill becomes three-eighths, changing the sign while the forecasts and outcomes are unchanged. The checker rejects that value. A favorable score can therefore be an artifact of choosing an inappropriate comparator, even when the arithmetic is otherwise correct.
Three evidence records and separate axes¶
| Record | Method and evidence | Scope and status |
|---|---|---|
| Research | Authored finite argument and explicit counterexample, 12 October 2026 | Analytical; limited to the definitions above |
| Reference | My execution, with AI assistance, of the independently designed public teaching case calibration, 12 October 2026 |
Fault-tested; externally reproducible with the commands below; no independent reproduction claimed |
| Production | None for this proposition | No product conformance inferred |
Engineering maturity: reference_implemented, referring only to the disclosed teaching case. Assurance: analytical, fault_tested, externally_reproducible, each restricted to its evidence record. These labels are independent dimensions, not a score or a global verification flag. The exact machine-readable record is in the public evidence catalog. My own runs do not count as independent review.
Limitations and open obligations¶
This four-case demonstration does not measure calibration in customers, establish causal benefit, or validate an incentive mechanism. Forecast selection, missing outcomes, adversarial reporting, cohort shift and adaptive stopping need their own methods. Experience transfer and human–LLM elicitation require empirical study; the scoring calculation is not such a study.
Prior art and contribution limit¶
Proper scoring rules have an established literature. Their properties do not automatically transfer to composite reputations or economic rewards. This case contributes a transparent comparator check and identifies the limit of what arithmetic alone can establish. See the primary reference and the comparison guide. No priority or superiority claim follows from this example.
Reproduction and falsification¶
From the successor public source package, use CPython 3.12.14 and no external packages:
python examples/investigations/check.py --case calibration
python examples/investigations/check.py --case calibration --mutant
The first command must exit zero, with no positive violations and at least one negative violation. The second intentionally exits one and prints an actual violated case. It is a deliberate fault result, not an installation failure. Source and reproduction contract identify the exact population and scope.
Submit a counterexample, a missing assumption or a prior-art correction through the review guide. Include GA-SYN-012, edition 2026.10.9, the command, input and output, and whether the objection concerns the model, implementation or proposed product relationship. Do not send credentials or confidential customer material.