Coding agent harnesses add cost, not pass rate
Trending storyDeveloping
Coding agent harnesses add cost, not pass rate
1 reports1 sourcesUpdated 2 hours ago
What you need to know
AI summary
A study by Liu and Han found that swapping coding agent harnesses shifts SWE-bench Verified results about as much as simply rerunning the same setup, while adding cost without improving pass rate. The finding suggests harness choice has limited effect on benchmark outcomes relative to run-to-run variance.
Generated by AI from the reporting · updated 2 hours ago
Latest updateOct 24 09:21
Study finds coding agent harnesses add cost but not pass rate on SWE-bench VerifiedTimeline
Follow the reports to see every side of the story.
Oct 24
- The DecoderStudy finds coding agent harnesses add cost but not pass rate on SWE-bench Verified
A study by Liu and Han found that swapping coding agent harnesses shifts SWE-bench Verified results about as much as simply rerunning the same setup.
Heat trend for this story
Not enough continuous data yet to chart a trend.