Functional correctness
How completely the patch satisfies the task, its contract, and expected observable behavior. Wrong behavior or incomplete requirements lower the score.
Forty-four coding-agent runtime routes received the same retry-policy repair task. Results compare public and objective correctness, four-dimensional manual review where available, solution time, provider-reported solution cost, and access channel.
Test assignment
The model received a small retry helper with deliberately seeded bugs. Its job was to make the smallest clear repair while keeping both exported functions unchanged and modifying only the allowed source file.
Retry-After either as whole, unsigned decimal
seconds or as a valid HTTP date, while rejecting signed,
fractional, hexadecimal, exponent, empty, and non-string values.
nowMs,
never return a negative delay, and reject an invalid date or
non-finite current time.
attempt to be a positive integer and
nowMs to be finite.
Retry-After override normal backoff;
otherwise use exponential delays of 1, 2, 4, 8 seconds, and so on.
Where a manual expert review is recorded, each criterion receives a score from 0 to 10. Higher scores indicate stronger performance; N/A means that the patch has not yet received this separate review.
Review policy. GPT-5.6 Sol is the single canonical expert reviewer to keep benchmark costs controlled. Independent Gemini cross-checks broadly agreed with its scoring. Claude Opus 4.8 was removed from future judging after repeatedly missing seeded hidden defects while evaluating candidate patches.
Independent verification. The test results, submitted patches, evaluation artifacts, and testing pipeline are publicly available. You can inspect the evidence, reproduce the process, and make your own independent assessment.
How completely the patch satisfies the task, its contract, and expected observable behavior. Wrong behavior or incomplete requirements lower the score.
How robustly the patch handles invalid, boundary, and hidden cases without brittle behavior. Passing happy-path or public tests but failing realistic edge cases lowers the score.
How readable, simple, idiomatic, and easy to reason about the implementation is. Unnecessary complexity, opaque logic, or hard-to-maintain code lowers the score.
How well the patch limits itself to necessary, focused changes within allowed file and task boundaries. Unrelated edits, overengineering, or forbidden and out-of-scope file changes lower the score.
Overall is the arithmetic mean of the four scores. Rank is ordered by status, then Overall descending, solution time ascending, and solution cost ascending.