Proof
Every number on this site comes from a benchmark you can re-run. This page shows the numbers, the baselines they are compared with, and what they do not mean.
The engine is faithful to the law
Before judging anything, the rules engine was checked against USDA's own answers Agreement with USDAWhat it is. On every in-scope audited household, our engine's benefit compared with the benefit USDA's reviewers calculated.Why it matters. It shows the rules engine is faithful to the law before we use it to judge anything..
Where the remaining mismatches come from
standard_deduction (Illinois records $7 below the federal table): 410medical_deduction (state standard medical deduction): 5net_income: 2shelter_deduction: 1
Excluded: SSI Combined Application Project (state-specific benefit) (874); outside 48 States and DC (different parameters) (1,592); Minnesota Family Investment Program (separate calculation) (98).
Seeded bugs on real households
Six realistic bugs, each a patch to the real engine source (so each broken release is real code with its own digest). Each shipped in one monthly release and was fixed six releases later. The earliest harmed household played the appeal. Then bisect, replay and remedy ran with no further input. Ground truth was computed separately by comparing every decision with the reference engine.
| Bug | Harmed | Bisect | Recall / precision Recall and precisionWhat it is. Recall is the share of harmed households found. Precision is the share of flagged households that were really harmed.Why it matters. Both are 1.0 here because replay re-runs the same decisions exactly. That depends on the ledger being complete, which the hash chain protects. | Restored | To a human | Appeals would find Waiting for appealsWhat it is. What happens today. Each harmed household is found only if it appeals. We apply real appeal rates to the harmed count.Why it matters. It is the fair comparison. Lookback should be judged against the current practice, not against nothing. | Analyst query precision Analyst queryWhat it is. How claims systems do mass reprocessing today. An analyst selects records by reason code and date range, then re-runs them.Why it matters. It over-selects and under-selects. Precision here is the share of selected households that were actually harmed. |
|---|---|---|---|---|---|---|---|
| B1 Shelter deduction capped for elderly or disabled households 7 CFR 273.9(d)(6) | 1,983 | correct, 5 probes | 1 / 1 | $106,427 1,983 hh | 0 | 228 or 4 | 9% |
| B2 Gross income test applied to elderly or disabled households 7 CFR 273.9(a) | 41 | correct, 5 probes | 1 / 1 | $4,342 41 hh | 0 | 5 or 0 | 100% |
| B5 Earned income deduction dropped 7 CFR 273.9(d)(2) | 4,411 | correct, 5 probes | 1 / 1 | $536,903 4,411 hh | 0 | 507 or 9 | 21% |
| B6 Minimum benefit not applied to one and two person households 7 CFR 273.10(e)(2)(ii)(C) | 1,983 | correct, 5 probes | 1 / 1 | $42,154 1,983 hh | 0 | 228 or 4 | 9% |
| B7 Household contribution rounded to the nearest dollar instead of up 7 CFR 273.10(e) | 5,171 | correct, 5 probes | 1 / 1 | $0 0 hh | 5,171 | 0 or 0 | 27% |
| B3 Household exactly at the gross income limit rejected 7 CFR 273.9(a) | 0 | 0 flagged of 21,211 replayed | |||||
| R0 Refactor only: rename a local variable (negative control) none | 0 | 0 flagged of 21,211 replayed | |||||
"Appeals would find" applies two real appeal rates to the harmed count: 11.5% (Medicare Advantage, 2024) and 0.2% (Colorado SNAP, FY2019). B7 is an overpayment bug: nothing is restored, and every finding goes to a human. B3 harmed nobody because no sampled household sits exactly on the limit; that is a finding, not a gap.
Cascade: a wrong certification harms every month after it
A SNAP benefit is decided at certification and paid every month of the period Cascade replayWhat it is. Following one household forward in time, because a benefit is set at certification and paid every month after.Why it matters. A wrong certification harms every month that depends on it. Replaying one month at a time misses most of the money.. Replaying one month at a time finds the same households but misses most of the money.
| Bug | Households | Replaying each decision alone | Carrying corrected certifications forward |
|---|---|---|---|
| B1 Shelter deduction capped for elderly or disabled households | 930 | $54,644 | $203,671 |
| B2 Gross income test applied to elderly or disabled households | 21 | $2,245 | $7,924 |
| B5 Earned income deduction dropped | 4,428 | $542,460 | $1,783,884 |
| B6 Minimum benefit not applied to one and two person households | 972 | $21,005 | $76,914 |
| B7 Household contribution rounded to the nearest dollar instead of up | 3,603 | -$3,625 | -$12,369 |
Months beyond a household's sampled month are constructed by carrying its circumstances forward and are labeled as such. ConstructedWhat it is. Data we built for the demo, labeled as such. The hearing decision is modeled on published New York decisions; months beyond a household's sampled month carry its circumstances forward.Why it matters. Headline numbers that depend only on real sampled data are reported separately from numbers that use constructed timelines.
The live run
A bug nobody here planted
rulespec-us is a public, open-source encoding of US benefit rules. Its FY2026 SNAP parameters carry a start date and no end date, and no FY2027 module exists, so for October 2026 onward it silently returns last year's amounts (issue #1394, open since September 24, 2026). We fetched their published values at run time, confirmed they match the USDA memo exactly, used them as the deployed decider for October 2026 on every in-scope audit household, and replayed against the FY2027 values the law requires.
The number is large because the bug is universal: a missed cost-of-living adjustment touches almost every household, by the size of the increase. This is exactly the failure Lookback's parameter loader refuses (a month with no parameters is an error, never last year's numbers), the monitor warns about a month ahead, and the replay gate blocks before merge. If it had shipped, replay would list every household and the remedy engine would restore the difference.
Every month a bug stays unfound
Under 7 CFR 273.17, restoration reaches back at most 12 months from discovery The 12-month limitWhat it is. Under 7 CFR 273.17, restoration reaches back at most 12 months from the date the agency discovers the loss.Why it matters. Every month a bug goes undiscovered is a month of benefits lost for good. Finding bugs faster is money.. Harm grows every month; after 12 months, every further month can never be restored. These are the dollars permanently lost, nationally, by how long each bug goes undiscovered.
| Bug | Harm per month | Lost for good at 12 months | at 18 months | at 24 months |
|---|---|---|---|---|
| B1 Shelter deduction capped for elderly or disabled households | $141,661,004 | $0 | $849,966,024 | $1,699,932,048 |
| B2 Gross income test applied to elderly or disabled households | $1,829,579 | $0 | $10,977,474 | $21,954,948 |
| B5 Earned income deduction dropped | $510,213,552 | $0 | $3,061,281,312 | $6,122,562,624 |
| B6 Minimum benefit not applied to one and two person households | $44,189,620 | $0 | $265,137,720 | $530,275,440 |
| EXT rulespec-us #1394 (stale FY2026 values in FY2027) | $324,547,206 | $0 | $1,947,283,236 | $3,894,566,472 |
Lookback's live run found its bug in the pipeline that merged the fix, with zero months lost. Waiting for appeals finds a fraction of the households, months later.
By state
The same bug (B5, earned income deduction dropped) by state, weighted, for one month, next to each state's FY2025 payment error rate and the share of all benefit costs it will owe from FY2028 under the One Big Beautiful Bill Act if that rate holds.
| State | Households harmed, month | Underpaid, month | FY2025 error rate | Cost-share tier from FY2028 |
|---|---|---|---|---|
| California | 560,233 | $58,510,878 | not quoted | above 6% (rate not quoted; 41 states plus DC were) |
| Texas | 379,179 | $45,903,263 | not quoted | above 6% (rate not quoted; 41 states plus DC were) |
| Florida | 345,130 | $36,912,537 | not quoted | above 6% (rate not quoted; 41 states plus DC were) |
| Pennsylvania | 248,900 | $29,456,939 | not quoted | above 6% (rate not quoted; 41 states plus DC were) |
| Illinois | 219,337 | $26,274,166 | not quoted | above 6% (rate not quoted; 41 states plus DC were) |
| New York | 222,423 | $25,588,227 | not quoted | above 6% (rate not quoted; 41 states plus DC were) |
| Michigan | 166,590 | $22,021,928 | not quoted | above 6% (rate not quoted; 41 states plus DC were) |
| North Carolina | 166,402 | $20,239,494 | not quoted | above 6% (rate not quoted; 41 states plus DC were) |
| Georgia | 149,013 | $16,598,614 | 15.21% | over 10%: pays 15% |
| Ohio | 138,091 | $15,582,004 | not quoted | above 6% (rate not quoted; 41 states plus DC were) |
| Massachusetts | 104,162 | $12,750,469 | not quoted | above 6% (rate not quoted; 41 states plus DC were) |
| Arizona | 87,734 | $12,269,493 | not quoted | above 6% (rate not quoted; 41 states plus DC were) |
Weighted with USDA FY2024 QC sample weights (FYWGT); federal rules only, 48 States and DC. Error rates: FY2025, USDA June 2026, as quoted by Ballotpedia; states not quoted are shown without a rate. A bug that underpays households does not by itself raise a state's payment error rate in its favor: underpayments count as errors too.
What it means nationally
USDA publishes a sample weight for each audited household: how many real households it stands for in an average month. Applying those weights turns sample counts into national estimates.
| Bug | Sample households harmed | Households nationally, average month | Underpaid nationally, per month |
|---|---|---|---|
| B1 Shelter deduction capped for elderly or disabled households | 1,983 | 2,260,626 | $141,661,004 |
| B2 Gross income test applied to elderly or disabled households | 41 | 16,002 | $1,829,579 |
| B5 Earned income deduction dropped | 4,411 | 4,374,830 | $510,213,552 |
| B6 Minimum benefit not applied to one and two person households | 1,983 | 2,058,114 | $44,189,620 |
| B7 Household contribution rounded to the nearest dollar instead of up | 5,171 | 4,938,931 | overpayment, to a human |
Point estimates from the fiscal-year weights (FYWGT). Standard errors need the separate 107 MB replicate-weight file, which is not loaded. The live run's bug, weighted the same way: about 677,879 households and $12,037,202 a month.
Monitoring, every day
A scheduled pipeline verifies the production ledger, replays every decision through the deployed engine, and compares the newest month's reason codes with the trailing months. Last run: no findings; 21,211 decisions across 6 months, 9 adverse findings awaiting a human, PSI 0.000. A parameter file that expires without a successor is caught here before it pays anyone wrong.
Energy and carbon (SCI)
Software Carbon Intensity per household remediated, from this project's own pipeline durations. The design choice that matters: replay is deterministic code, so finding 413 households costs about the energy of one CI job rather than one model call per decision.
Coefficients and caveats
- Energy for CI jobs is derived from measured job durations and a per-vCPU power coefficient; it is not metered.
- LLM energy per token is an order-of-magnitude public estimate; the comparison's conclusion does not depend on its exact value.
- Grid intensity is a regional average; a range of 0.35 to 0.50 changes totals proportionally.
- The design choice that matters: replay is deterministic Python, so finding 413 households costs about the energy of one CI job, not one model call per decision.
Coefficients: W_per_vCPU = 2.3; grid_kgCO2e_per_kWh = 0.42; embodied_kgCO2e_per_vCPU_hour = 0.003567; llm_Wh_per_1k_tokens = 0.4. Formula: SCI = ((E x I) + M) / R, Green Software Foundation.
What the numbers do not mean
- Recall and precision are 1.0 because replay re-runs the same decisions exactly. That depends on the ledger being complete and replayable, which is what the hash chain and the sample re-runs protect. It is a property of the method, not a feat of detection.
- Counts are unweighted sample households, one decision each. They are not national estimates.
- Federal SNAP rules only, 48 States and DC. State options are not modeled; the exclusions above say which households that affects.
- The demo ledger's months are the QC sample months shifted forward 21 months so they fall inside the 12-month restoration window. The shift is recorded in each record's provenance.
Re-run it yourself
git clone https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/38491653/showcase.git && cd showcase
python -m venv .venv && .venv/Scripts/activate
pip install -e ".[test]"
pytest # 82 tests
python -m bench.qc_agreement # needs data/raw/qcfy2024_csv.zip (see data/raw/README.md)
python -m bench.lookback_bench
python -m bench.cascade_bench
python -m bench.site_data # regenerates the numbers on this siteSite build 7cde0ca, data generated 2026-10-09T04:23:14+00:00.