I would have thought that given a test suite, agents would probably not claim a bug was fixed when the test still failed. However, Orkes was able to run the generative test suite I was writing through this process, and we had to go back and forth several times on bugs which were supposedly fixed but weren’t.
Part of that might just be normal human struggle to keep up with a high volume of complex bugs. I normally find a handful of serious bugs in a 4-8 week collaboration, and here I found 10-20 per week. Likewise, I normally test just a few versions, and here we’d have several builds to test each week. I’m still waiting on Orkes to publish the repo with the issues we found, but once that goes live you can see both of us made mistakes about what versions had which bugs, or exactly what reproductions look like.
Another challenge is that ParticleDB was so broken that the tests would fail for lots of reasons. To a human without intuition, or maybe to an LLM, it might be hard to tell whether a bug was fixed. Our most productive test, compare, also produced a fair bit of false positives–places where Postgres and ParticleDB behavior diverged in harmless ways. Easy for a human to sort through, but apparently really hard for LLMs to cope with.
Agents will also just… flat-out lie to you about what the tests say, so that doesn’t help!
I would have thought that given a test suite, agents would probably not claim a bug was fixed when the test still failed. However, Orkes was able to run the generative test suite I was writing through this process, and we had to go back and forth several times on bugs which were supposedly fixed but weren’t.
Part of that might just be normal human struggle to keep up with a high volume of complex bugs. I normally find a handful of serious bugs in a 4-8 week collaboration, and here I found 10-20 per week. Likewise, I normally test just a few versions, and here we’d have several builds to test each week. I’m still waiting on Orkes to publish the repo with the issues we found, but once that goes live you can see both of us made mistakes about what versions had which bugs, or exactly what reproductions look like.
Another challenge is that ParticleDB was so broken that the tests would fail for lots of reasons. To a human without intuition, or maybe to an LLM, it might be hard to tell whether a bug was fixed. Our most productive test,
compare, also produced a fair bit of false positives–places where Postgres and ParticleDB behavior diverged in harmless ways. Easy for a human to sort through, but apparently really hard for LLMs to cope with.Agents will also just… flat-out lie to you about what the tests say, so that doesn’t help!
score 1 · canonical
My favorite is when the tests are entirely composed of test doubles.
score 1 · canonical