Yes. Tests are just code which should be heavily constrained to be:

• simple
• short
• linear
• WET not DRY
• deterministic
• explanatory rather than algorithmic
• non tautological
• descriptive of the failure mode (i.e. don’t assert(foo.is_empty) instead assert_eq(foo, []))
• covering the useful branches of the implementation
• usually correct (exception: characterization tests used to note drift prior to fixing a bug)

The tests that come out of the default models sometimes tend to not quite fit this, but most frontier models respond pretty well to steering to get this right, so invest in a good set of rules that match your domain on this. There’s lots of simple rules you can build up to write better tests in any language.

The thing about tests and LLMs is that you can use just two of the things spec + impl or spec + test in isolation and have the model reason over them for whether they’re correct. This is a sort of triangulation type thing that you do as a developer and it’s reasonable to do with an LLM too and not just treat it like a black box that summons code into being in one incantation.

The other thing you can do is have the LLM quickly go through the process of testing the Red/Green steps in TDD (whether pre-testing or post-testing) to ensure that a failure shows up somehow and is correctly indicating what went wrong.

LLMs are better at this than humans, assuming they get decent inputs upfront.

score 1 · canonical