For the Bluesky backend, we essentially do a SELECT * FROM blocks, mutes, list items, etc. every time one the processes starts, and add all the items to the bloom filter

If you’re constructing a bloom filter but not querying it yet, I think you can split that across many threads without contended locks or atomics, by having each thread have its own private bitmap and setting bits in that. Then when all threads have finished inserting items, you bitwise OR together all the bitmaps.

There’s also a real time atproto firehose consumer inserting items in to bloom filters (even during process startup), so there is a race condition if you’re not careful.

Add another bitmap for events that are showing up during process start and bitwise OR that in at the end, too.

Alternatively if you happen to have two bitmaps and no opportune time to merge them, you could keep querying both and returning 1 where either bitmap has a 1.

score 1 · canonical

Nice! Condensed:

• “shredding” is copying fields out of the json objects into their own columns so you can query them fast without parsing every json object
• the first thing they tried was “static shredding” where they do this for a hardcoded list of fields
• the new nice thing is “dynamic shredding” where they pick which fields to copy out based on observed usage patterns, at the time they build the parquet files.

score 1 · canonical

I haven’t used Deno for anything much but I’m very grateful that the competition spurred some advances in node

score 1 · canonical

I hate the entire existence of hooks for other reasons so this seems like a good push for me to configure git to never run them anywhere ever.

score 1 · canonical

Good news: it can’t be nearly as bad as the py2 → py3 transition was. Even if we all decided to switch to a completely new version control system like Pijul or Mercurial or Darcs or something, it wouldn’t be nearly that bad. We already did it multiple times (e.g. mass exodus from svn to various DVCSes, then to git) and it was never a huge deal.

You can convert one repo at a time. The vast majority of software builds don’t interact with the VCS tools of their dependencies other than “how do I download the source code given a URL”. Build tools can add support for new VCS types without breaking existing users of the existing ones.

The python 2→3 transition had exceptionally nasty network effects: users couldn’t move to py3 until all their dependencies were available on it. The big frameworks users depended on couldn’t simply move to py3 unless most of their users had moved: they had to stay py2 compatible. For a few years (until python 3.2 was released) there wasn’t any good way of writing python code that would parse and function identically in both py2 and py3. (The 2to3 translation tools were imperfect and in practice everyone needed a way to write polyglot code instead.)

score 1 · canonical

It’s way nicer but also if you’ve made an API that looks like this, I think you may as well go a little further and make it create parser combinators like Parsec’s style or LL(1-or-more) parsers because they tend to be very readable too for exactly the same reason…

score 1 · canonical