I used to be pretty opposed to this, but the models have gotten significantly better and nowadays I probably let the LLMs write both the code and the tests about half the time. The LLMs all do seem to go a bit hog-wild with low-value tests, especially “prove that the code no longer works the way it used to” tests, but (and this is the big change in recent models) you can successfully instruct them to not do that.

Using something like the Superpowers plugin helps a lot since it makes the models use a TDD-style workflow and gets them to be more rigorous about gathering requirements before even starting in on the code design.

A common flow for me is to have an initial back-and-forth to define the problem and account for all the corner cases the tool can dig up, then review its high-level design document, then let it rip on both the code and tests, and finally, tell it to go trim down and consolidate its tests. On a change I’m working on now, that last step caused my LLM to get rid of a solid 40% of its original test code without losing much utility.

Another important thing I do is review the code and the tests myself before anyone else on my team looks at them. If there are low-value or overly-verbose tests, I clean them up (usually by hand, sometimes by LLM) before sending the change out for code review.

score 1 · canonical

The other day I reviewed a PR for one of my projects that was vibe-coded. The PR comprised a single commit with a small fix and lots of additional tests. The single commit means there’s no (easy) way to really make sure all those new tests would fail without the fix.

I think it would have been better to have the agent first write failing tests, commit that, then work on the fix and commit it separately. While I don’t vibe-code myself, I think following a strict TDD methodology can make a big difference in this respect.

score 1 · canonical

For throw-away code (HTML dashboards, code that generates graphs for slides, etc.) I don’t bother looking at the code at all. The LLM writes tests and I presume it’s useful to avoid regressions, but again - I don’t really look.

score 1 · canonical

To me a lot of LLM tests feel like “scaffolding” - they are not written like you would write a test you intend to keep, but may have helped it validate something that does not need to continually be validated once the code is done.

As such, I have arranged things such that most of the tests it writes wind up in a directory that is, by default, gitignored, and it requires more justification to put them where they’ll be checked in.

score 1 · canonical

You can’t do both in a single session, because LLMs will write weaksauce tests to make its code look good.

It works better if you start a new session and give it a goal to find bugs in the code and prove that via tests.

score 1 · canonical

I’m still in the “haven’t used LLM’s to generate any code” camp. The most I’ve used a LLM for was to generate a backstory for a D&D character. I avoid that plague as much as I can.

score 1 · canonical

Yes. Tests are just code which should be heavily constrained to be:

• simple
• short
• linear
• WET not DRY
• deterministic
• explanatory rather than algorithmic
• non tautological
• descriptive of the failure mode (i.e. don’t assert(foo.is_empty) instead assert_eq(foo, []))
• covering the useful branches of the implementation
• usually correct (exception: characterization tests used to note drift prior to fixing a bug)

The tests that come out of the default models sometimes tend to not quite fit this, but most frontier models respond pretty well to steering to get this right, so invest in a good set of rules that match your domain on this. There’s lots of simple rules you can build up to write better tests in any language.

The thing about tests and LLMs is that you can use just two of the things spec + impl or spec + test in isolation and have the model reason over them for whether they’re correct. This is a sort of triangulation type thing that you do as a developer and it’s reasonable to do with an LLM too and not just treat it like a black box that summons code into being in one incantation.

The other thing you can do is have the LLM quickly go through the process of testing the Red/Green steps in TDD (whether pre-testing or post-testing) to ensure that a failure shows up somehow and is correctly indicating what went wrong.

LLMs are better at this than humans, assuming they get decent inputs upfront.

score 1 · canonical

I do have LLMs generate both implementation and tests. But I spend a lot of time thinking about what needs to be tested and how, to give me confidence the implementation is correct. I.e. I don’t give the LLM free reign in deciding what and how to test.

I’m actively experimenting with methods for getting verification right at a pace faster than I can personally guarantee all of the implementation. Things like STAMP/STPA, mutation testing, fuzzing etc. that aim to make verification sliyghtly more algorithmic could be fruitful ways forward. But it’s still early days and I have to step in and manually correct a lot.

score 1 · canonical

I really dislike this as well, and I’m not a supporter of LLMs, but some humans can also do this without any LLM. Maybe you can simply say that it’s over-engineered and that the tests are bad. I think it’s a valid reason to refuse to merge a PR.

But to answer your question, I have used LLMs to write both tests and code a few times…

score 1 · canonical

Sometimes code & unit tests, then I write the integration & e2e tests. It depends on the code & domain.

score 1 · canonical