Discussion about this post

User's avatar
Abhishek Pan's avatar

A 0.50 completion score on runs that passed 87% of the time means the controller kept asking for verify steps on finished work, which lines up with the 19% cost bump. Good ranking with bad calibration is easy to miss offline, so I'd plot predicted confidence against actual pass rate before letting any small model decide when an agent stops.

Ready for more?