Functional correctness
How fully the patch restores the documented ledger behavior.
Clean-room benchmark · Bug fixing
Bug-fixing test: repair ledger reconciliation
This test measures whether a coding model can diagnose an existing accounting defect and repair it without changing the contract or widening the patch.
76 tested runtime routes
76tested routes
75completed patches
57objective passes
0unavailable routes
Test assignment
Fix duplicates, sales, refunds, and invalid ledger rows
The model receives a small ledger reconciler with seeded defects. It must understand the documented behavior, find the incorrect implementation, and make the smallest correct repair while preserving both exported functions and the result format.
What the repaired code must do
- Validate every row and count unusable entries without crashing.
- Process only the first occurrence of a transaction ID and count every later occurrence as a duplicate, including duplicate refunds.
-
Parse non-negative dollar amounts exactly, accepting optional
$and correctly grouped commas without rounding malformed values. - Group totals by account, currency, and the row's UTC date.
- Subtract a refund from its previously processed sale's account and currency; count an unmatched refund as an orphan.
- Keep totals sorted and count as processed only rows that actually affect them.
How each reviewed solution is scored
Each criterion receives 0–10. Public and objective checks remain separate from the review score.
Review policy. GPT-5.6 Sol is the single canonical expert reviewer to keep benchmark costs controlled. Independent Gemini cross-checks broadly agreed with its scoring. Claude Opus 4.8 was removed from future judging after repeatedly missing seeded hidden defects while evaluating candidate patches.
Independent verification. The test results, submitted patches, evaluation artifacts, and testing pipeline are publicly available. You can inspect the evidence, reproduce the process, and make your own independent assessment.
Reliability / edge cases
How safely it handles malformed rows, exact money, duplicates, dates, and orphan refunds.
Maintainability / clarity
How readable, direct, and easy to reason about the repair is.
Scope discipline
How strictly the patch stays within the allowed file and avoids unrelated work.
Overall is the arithmetic mean of the four scores.