So, the problem we’re trying to solve here is that JSON parsing is a bottleneck. Their solution is to do some cheaper pre-processing so they have less parsing to do. But then I have to ask:

Why are we using a slow text format to begin with?

I understand that not all past decisions can be reversed, and sometimes there’s no choice because of reasons¹. Still, before we even contemplate solving a problem, I believe we should think whether we could avoid it entirely.

In this case: how about using a custom binary format instead of JSON? You may not even have to roll your own if something like MessagePack is fast enough despite being pretty generic. And if you do roll your own, chances your format could be simple enough that it wouldn’t take more effort than implementing raw filtering.

[1]: Like, text is more readable and easier to debug when all your tools are text based. We started with JSON in our prototype and now our “production grade prototype” that calls itself an application is built around JSON and we’re stuck.

score 4 · canonical