lobsters score: 2 · tags: python
Fun fact for the haters: Daniel lemire is one of the original authors of simdjson
score 1 · canonical
I came across one of his old blog posts a few months ago, and I thought ‘wait, that doesn’t sound right, who is this guy?’. Then I reread the blog name and thought ‘from SIMDJSON?’, did more research, and confirmed that, no, he was right, I was wrong. As often happens.
score 1 · canonical
I am begging Python bloggers to understand what FFI is. You’re not doing it in Python, you’re doing it in C++ with a Python interface.
score 1 · canonical
I’m sympathetic to the general version of this complaint, but Lemire is one of the two authors of the original SIMDJSON paper in C++.
This is not the case of a random Python blogger being unaware that the packages he’s using weren’t themselves programmed in regular Python. He’s measuring how well the new free-threaded interpreter can work with a highly-parallel extension. Turns out, pretty well, and it’s a boon to the ecosystem.
But again, I am sympathetic, and ‘write the library in another language and then make a Python interface for it’ is my preferred way to interact with Python these days.
score 1 · canonical
I’m reacting more to the title than the substance, I think. I agree that what the post is actually talking about is useful information: using a Python interface to SIMDJSON doesn’t kill the performance due to GIL. That’s a useful thing to test and write about.
But “Parsing JSON … in Python” very clearly establishes an expectation that you are talking about parsing JSON in Python, which is contrary to the post. It’s a bait-and-switch.
score 1 · canonical
It seems more charitable to assume that Python developers in fact know that they’re writing Python extensions, and that “in Python” refers to using any given piece of native code with Python’s memory/GIL/object models.
score 1 · canonical
So, can you parse JSON at tens of gigabytes per second in Python?
I dunno, it’s a bit disingenuous to say yes to that when the actual hard work of parsing is done by simdjson.
“Can you parse JSON at tens of gigabytes per second if you use a library that can parse JSON at tens of gigabytes per second?” is a much less clickbaity title, I’ll concede.
score 1 · canonical
I think it’s pretty normal in the Python community to use “in Python” to refer to any library code that interacts with Python’s runtime, which would include native extensions. The hard work of parsing JSON in Python’s stdlib is done in C, after all.
(This is fair/reasonable IMO insofar as extensions do pay a significant performance cost, regardless of being written in native code.)
score 1 · canonical
The interesting part is how to design a Python API for a fast library that allows the programmer to do something useful without reducing the performance to Python levels. Or rather, it would be interesting if there was a discussion of how they did it beyond a vague note about lazy object creation. And wouldn’t it be more interesting to include pysimdjson in the benchmark, since I guess it’s the closest to fastsimdjson?
score 5 · canonical
Very nice. I also continue to be impressed how much performance msgspec gets as well.
score 1 · canonical