lobsters score: 2 · tags: databases, testing
I think it’s pretty unfortunate that people will take (and are already taking) this report to mean that coding with LLM assistance is bad, instead of taking the report to mean that the team at Orkes built a bad product that they didn’t actually test at all during development. This report will be widely shared among the anti-AI contingent online as if it is validation of the chip on their shoulder, even though people wrote bad software long before LLMs ever existed.
score 1 · canonical
Dang. I’ve never seen Jepsen link to a Tiktok meme before.
…wow this is really appropriate to post on Halloween.
score 1 · canonical
This is the most devastating Jepsen report I’ve ever read, and I will personally be hammering the entirety of section 4.5 (“LLM Tradeoffs”) to our company doors. Too many good quotes to pull, but this from section 4.3 is perhaps the most emblematic:
Most databases have a test suite which exercises at least basic examples of each feature, and it is customary to run those tests before release, if not continuously. Where were those tests for ParticleDB? How does one add support for NOT NULL constraints without a test which tries to insert at least one NULL value? What development process adds a CHARACTER(10) type without trying to write a string of (e.g.) five characters?
It does raise the question of why Orkes engaged Jepsen in the first place; like hiring Michelangelo to paint your toilet. How could they be so utterly unaware of the most basic flaws in their product? The impression one takes away from the report is of an entire team deep in AI psychosis, experiencing their work (?) solely through agents, wholly out of touch with reality.
(Also great to see Kyle giving technical writing its due, another all too common victim of the mad dash for LLMs.)
score 4 · canonical
It does raise the question of why Orkes engaged Jepsen in the first place;
If they’re sloppy enough to miss the most basic tests, then they have an (unwarranted) high confidence in their abilities. They just don’t get that they’re incompetent.
Where were those tests for ParticleDB?
It’s 2026. It’s not hard to write 1000’s of tests. But you have to have the idea that tests are useful.
Hey AI, please look at feature X, Y, and Z of PostgreSQL. Write a bunch of tests to validate the behavior.
… some churn later…
OK, AI, thanks! Now use those tests to validating that my PostgreSQL compatibility code is actually compatible. If my code doesn’t pass the tests, then fix the code, not the tests.
The impression one takes away from the report is of an entire team deep in AI psychosis,
I agree. They’re deep in the throes of “Look at all of the code we’re writing!” via AI prompts. They haven’t looked at the functionality, written tests, or done the most basic engineering around the subject.
score 1 · canonical
This is catastrophic. Who ever read this report will never ever use ParticleDB or any Orkes product. I am not quiet even sure if I would invite a candidate to an interview with that in their CV. The whole reading experience for me was head shaking and eye rolling.
score 1 · canonical
I will add this entry to my memorable quotes register: “like hiring Michelangelo to paint your toilet”. It does feel that software is getting a bit out of control and that some of us are just sailing through any kind of due diligence. We need to learn restraint and stop implementing features we don’t test, or even use.
score 1 · canonical
That’s a disconnect I’ve seen a few times when a company engages external consultants before addressing known issues or asking internally for a review. Then again, from a purely business perspective… I’m not sure it’s that terrible? If you want to do some specific kind of review, don’t have the available hours internally, and don’t want more permanent staff - maybe letting the consultants raise some basics/duplicates during the engagement isn’t a bad outcome?
I had different thoughts about it in different situations. There’s a fuzzy threshold there between targeted money spending and madness.
score 1 · canonical
To this point–I do a fair bit of consulting for products that are still in their early stages. A lot of that is unpublished, but the Redis-raft and Capela reports are both on pre-release systems. I’ve found that phase of engagement can be fantastically productive–we can shake out bugs rapidly and give the team a regression test suite that they can use as they start rewriting prototype code, or adding new optimizations.
The thing I try to impress upon clients is that they should be honest about their public claims. If the software is experimental or pre-release, say so. If you have low confidence in its correctness, or don’t know exactly what semantics you’d like to guarantee, that’s OK too. All you gotta do is communicate that. :-)
score 2 · canonical
This wasn’t just “known issues”, the system didn’t work and hadn’t been tested; reading between the lines one could argue that it had only been run by the LLM. Taking this very public step with software in this state was terrible judgement and has put the product - and possibly the whole company - at risk.
score 2 · canonical
Note that the previous Jepsen report last week was for different (less broken) software from the same supplier. Since ParticleDB is intended to support Conductor, it makes sense that a full examination of Conductor should also cover ParticleDB; and since Conductor can use alternative databases (Jepsen tested with Postgres), it makes sense to test ParticleDB separately.
score 2 · canonical
Jepsen recommends users halt further deployment of ParticleDB and migrate to other databases as soon as possible.
That seems like the right conclusion, wow.
I have to wonder what leads a database developer to hiring Jepsen without first doing even fairly basic testing of their own database. Though I guess in this case it was probably “the LLM told them to”, and they didn’t even take the preliminary step of asking the LLM to do its own Jepsen-style analysis first (which obviously is nowhere near the same standard, but it sure sounds like it would have caught a lot of this stuff). Or maybe the LLM claimed that it had done so and they believed it?
Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work.
score 1 · canonical
I asked about this too, hoping to discuss it in the report. FWIW, I ran the same Postgres Jepsen test against ParticleDB–it took a little bit to get it to set up the database and connect, but once it did, it started screaming about every possible kind of safety violation. I’m not sure how they got it to pass!
score 2 · canonical
…“As with most Jepsen analyses, Orkes engineers offered detailed explanations of the causes and resolutions of the bugs we found in Conductor. However, Orkes seemed unable to explain the causes or resolutions of nearly every issue we found in ParticleDB.”…😬
I am wondering to what extent can this be mitigated (bugs) by relying on DST and tools like Antithesis?
Naively, based on vibes from Twitter: DB + AI + DST + Formal Methods == Bugs Vanished. I am curious on how far can we push this?
Though at the end of the day, I would have a hard time on trusting a DB built this way.
score 1 · canonical
Sure, DST would have been helpful in finding bugs. Even basic manual QA would have been helpful! Most of these issues just required starting a fresh node and typing a handful of SQL queries into the shell.
Understanding a bug is different than finding one. To understand the cause of a bug, you probably need to read some of the code involved, or at least have a solid mental model of the system. Until ~2025, clients generally needed to read code and understand a bug to fix it. With AI, you can just paste a bug report into Codex and say “fix”.
Anecdotally, that does work! ParticleDB is much safer now than it was when we began. I don’t think this kind of AI development is as robust a process as human understanding though. From working with a few clients who rely heavily on LLMs, bugs are often “fixed” but still happen, or a specific case would be resolved but the general pattern remained, or the fix would introduce even weirder behavior.
score 4 · canonical
To what extent do the bugs pop back up because the clients aren’t using generative/simulation testing/fuzzing?
I’ve been fairly blindly asking LLMs to write simulation tests as well as non-simulated tests (golang) and that definitely finds issues, but at minimum I need to review the code for potential issues and then the clanker writes tests and finds more issues. It feels a little bit like I’ve made the fastest whackamole machine.
score 1 · canonical
I would have thought that given a test suite, agents would probably not claim a bug was fixed when the test still failed. However, Orkes was able to run the generative test suite I was writing through this process, and we had to go back and forth several times on bugs which were supposedly fixed but weren’t.
Part of that might just be normal human struggle to keep up with a high volume of complex bugs. I normally find a handful of serious bugs in a 4-8 week collaboration, and here I found 10-20 per week. Likewise, I normally test just a few versions, and here we’d have several builds to test each week. I’m still waiting on Orkes to publish the repo with the issues we found, but once that goes live you can see both of us made mistakes about what versions had which bugs, or exactly what reproductions look like.
Another challenge is that ParticleDB was so broken that the tests would fail for lots of reasons. To a human without intuition, or maybe to an LLM, it might be hard to tell whether a bug was fixed. Our most productive test,
compare, also produced a fair bit of false positives–places where Postgres and ParticleDB behavior diverged in harmless ways. Easy for a human to sort through, but apparently really hard for LLMs to cope with.Agents will also just… flat-out lie to you about what the tests say, so that doesn’t help!
score 1 · canonical