lobsters score: 1 · tags: databases, distributed
I’ve experienced Kafka being used as a job queue at various places, and to me it typically felt like an awkward fit (or had lots of userspace code to deal with various replay scenarios).
I think I would have probably use NATS/jetsteam instead.
score 1 · canonical
I am not a fan of “use a database as a message queue” (and I spend every day living with the consequences of my predecessors doing exactly that in Oracle) but yes, NATS is so much easier than Kafka across just about all dimensions. I am a huge fan of NATS.
score 2 · canonical
What’s the issue with database as queue? Is it poor implementations? Too much traffic?
My general opinion is to use the existing database to avoid having to stand up a new system, but maybe I’m missing something.
score 1 · canonical
There’s nothing wrong with using the database for work queues if you’re careful and your needs are straightforward. It’s fine for relatively small volumes and simple systems and it’s excellent - maybe even indispensable - for transactional work. In fact I often use a database table as a transactional buffer between the database and the messaging system.
But NATS (for example) comes with a lot of features that you don’t get from a database table. For example it’s remarkably difficult to get message ordering guarantees from a Postgres database. So you can end up doing a lot of work to make it work properly, if it’s important to you (it is not always important)
There are lots of reasons why I prefer NATS to database tables, for example in message driven systems with many messages and many consumers the database can quickly become a bottleneck, and since most databases only scale vertically, this can paint you in a corner. Plus you generally need to poll for messages which ends up putting a parasitic load on the database.
At least in my (inherited) experience at $WORK, putting the queues in the database also leads to polling antipatterns which pull in a bunch of other stuff and become very expensive. I am forced to deal with the serious consequences of these problems every day so I am a bit sensitive to this topic!
I guess to summarise, an external broker creates a clear and decoupled boundary between the database(s) and the work queues, which helps prevent this stuff.
score 1 · canonical
This makes sense. I think I haven’t run into the more advanced cases in my career. NATS (plus or minus jetstream) seems cool–I would like to get an opportunity to work with it at some point.
score 1 · canonical