Current isolated benchmark · Retry-policy patch

Patch test: correctly interpret Retry-After

Forty-two coding-agent runtime routes received the same retry-policy repair task. Results compare public and objective correctness, four-dimensional manual review where available, solution time, provider-reported solution cost, and access channel.

42 tested runtime routes
42tested runtime routes
37completed candidate runs
4unavailable routes
1agent failure

Test assignment

Retry policy: correctly interpret Retry-After

The model received a small retry helper with deliberately seeded bugs. Its job was to make the smallest clear repair while keeping both exported functions unchanged and modifying only the allowed source file.

What the repaired code had to do

  • Accept Retry-After either as whole, unsigned decimal seconds or as a valid HTTP date, while rejecting signed, fractional, hexadecimal, exponent, empty, and non-string values.
  • For a date, calculate the remaining delay from nowMs, never return a negative delay, and reject an invalid date or non-finite current time.
  • Require attempt to be a positive integer and nowMs to be finite.
  • Let a valid Retry-After override normal backoff; otherwise use exponential delays of 1, 2, 4, 8 seconds, and so on.
  • Cap every delay at five minutes, including extremely large retry attempts, so the result always remains finite.
Allowed file: fixtures/phase2-v2/patch-retry-policy/src/retryPolicy.js

How each reviewed patch is scored

Where a manual expert review is recorded, each criterion receives a score from 0 to 10. Higher scores indicate stronger performance; N/A means that the patch has not yet received this separate review.

Functional correctness

How completely the patch satisfies the task, its contract, and expected observable behavior. Wrong behavior or incomplete requirements lower the score.

Reliability / edge cases

How robustly the patch handles invalid, boundary, and hidden cases without brittle behavior. Passing happy-path or public tests but failing realistic edge cases lowers the score.

Maintainability / clarity

How readable, simple, idiomatic, and easy to reason about the implementation is. Unnecessary complexity, opaque logic, or hard-to-maintain code lowers the score.

Scope discipline

How well the patch limits itself to necessary, focused changes within allowed file and task boundaries. Unrelated edits, overengineering, or forbidden and out-of-scope file changes lower the score.

Overall is the arithmetic mean of the four scores. Rank is ordered by status, then Overall descending, solution time ascending, and solution cost ascending.

Complete candidate results

Activate any column heading to sort; activate it again to reverse direction. N/A values remain last.

42 rows

Current retry-policy patch results for 42 tested runtime routes.