How much of a SWE-bench Verified score comes from the grader, the task and the scaffold rather than the model. Hidden fields (gold patch, tests) are never served.
SampleReproducible stratified sample of Verified tasks
ControlsGold patch must resolve, empty patch must not
Re-gradeCompare our verdicts with the reported ones
Leak scanDid agents read future git history?
Noise floorRank stability when tasks are dropped