Functional correctness
How completely the patch satisfies the task, its contract, and expected observable behavior. Wrong behavior or incomplete requirements lower the score.
Forty-two coding-agent runtime routes received the same retry-policy repair task. Results compare public and objective correctness, four-dimensional manual review where available, solution time, provider-reported solution cost, and access channel.
Test assignment
The model received a small retry helper with deliberately seeded bugs. Its job was to make the smallest clear repair while keeping both exported functions unchanged and modifying only the allowed source file.
Retry-After either as whole, unsigned decimal seconds or as a valid HTTP date, while rejecting signed, fractional, hexadecimal, exponent, empty, and non-string values.nowMs, never return a negative delay, and reject an invalid date or non-finite current time.attempt to be a positive integer and nowMs to be finite.Retry-After override normal backoff; otherwise use exponential delays of 1, 2, 4, 8 seconds, and so on.Where a manual expert review is recorded, each criterion receives a score from 0 to 10. Higher scores indicate stronger performance; N/A means that the patch has not yet received this separate review.
How completely the patch satisfies the task, its contract, and expected observable behavior. Wrong behavior or incomplete requirements lower the score.
How robustly the patch handles invalid, boundary, and hidden cases without brittle behavior. Passing happy-path or public tests but failing realistic edge cases lowers the score.
How readable, simple, idiomatic, and easy to reason about the implementation is. Unnecessary complexity, opaque logic, or hard-to-maintain code lowers the score.
How well the patch limits itself to necessary, focused changes within allowed file and task boundaries. Unrelated edits, overengineering, or forbidden and out-of-scope file changes lower the score.
Overall is the arithmetic mean of the four scores. Rank is ordered by status, then Overall descending, solution time ascending, and solution cost ascending.
Activate any column heading to sort; activate it again to reverse direction. N/A values remain last.
42 rows