lobsters score: 1 · tags: performance
Misses the main thing: slow languages don’t exist. Languages aren’t slow. Slow implementations exist, and can be replaced over time by faster implementations.
score 1 · canonical
I don’t think this is correct.
Changes to implementation can absolutely have big effects on performance within a single language. CPython gets faster with most new version as the interpreter and the standard library improve, and you can step out CPython into Jython, or IronPython, or JITted PyPy, and see very different performance characteristics while remaining recognizably the same language. We might even conceive of an OCaml-like dual interpreter/compiler that can do both.
But there are differences in language semantics which mere implementation details can’t bridge.
A language whose semantics include loose or dynamic typing will place restrictions on the implementation that will reduce performance. A language whose semantics relies on dynamic dispatch without the ability to monomorphize will see unavoidable overhead. A language which abstracts away memory allocation takes a lot of optimization tools away from the program author, etc.
Changes to core language semantics like these would be extremely disruptive even in a whole-version migration breaking backwards compatibility. Adding required static typing to a dynamic language or manual memory management to a previously GC’ed language would be such a large difference that you almost certainly can’t call it the same language before and after.
You can of course write extensions in faster languages, but at that point you’re deliberately stepping outside your language’s established semantics because you’ve acknowledged that they’re too restrictive or insufficiently powerful to achieve your goals.
So I don’t agree that slow languages don’t exist. By virtue of their core semantics and the abstractions they aim to provide to the program author, all programming languages limit the performance they can deliver even with an ideal implementation.
score 4 · canonical
A language whose semantics relies on dynamic dispatch without the ability to monomorphize will see unavoidable overhead.
I’m not sure there are languages that really can’t be monomorphized. In principle someone could implement such a dynamic language with a Self-style JIT that dynamically monomorphizes, so the code can be optimized to perform as well as static dispatch. But I suppose the JIT itself is unavoidable overhead :-)
Also, a dynamic language’s design can make it easier to monomorphize, by making it clear when the metalevel is being mutated (things like class definitions) and when those mutations take effect. But that kind of design is relatively rare: CLOS is designed that way, and I think Java too (tho Java relies more on dynamic reoptimization than CLOS).
A language which abstracts away memory allocation takes a lot of optimization tools away from the program author, etc.
Yeah, but I’d say memory layout is more important than allocation: if a program can control when subobjects are embedded in other objects and in arrays, and isn’t required to refer to them indirectly, then it’s easier to get better memory performance. This is one of the big advantages of Golang relative to Java. And project Valhalla, adding value types to Java, has taken many years of careful work to avoid being disruptive.
Data structure layout at a high level (how objects relate to each other, as opposed to the internal layouts of objects) is in almost all languages controlled by the programmer. Language semantics usually don’t allow data layouts to be changed by optimizers because it’s prevented by features like object identity and mutability and aliasing. So I think data layout is the primary thing that determines whether a language is “fast” in the sense of being able to get the most out of the hardware (assuming it isn’t held back by a slow implementation).
score 3 · canonical
I don’t believe e.g. Haskell typeclass instances can always be monomorphised:
pathological :: (Show t) => (Integer, t) -> String pathological = \case (0,x) -> show x (n,x) -> pathological (n-1,(x,x))Anybody who writes code like this is going to pay a cost in runtime performance, one way or another.
score 2 · canonical
It’s totally possible to write benchmarks or inspection testing that would require heavy (e.g. whole-program) optimization. You’re assuming that all Haskell will always exhibit GHC’s limitations.
So no, you cannot guarantee people will pay a runtime cost for something that is testable and preventable before runtime.
score 1 · canonical
I understand it’s a difficult case, but if it is pathological, wouldn’t we be using a outlier to generalize from, it we pretended that this program needs optimization? If these programs are very rare. I suppose it all rests on what “code like this” means…
score 1 · canonical
The author has a point that big design or algorithmic changes can matter a lot. But if you’re rewriting Ruby, Lua, Perl, PHP, or most Python into basically anything else, it’s going to be way fucking faster. The author doesn’t just leave this out, he actively eschews it. That’s wrong, and only explainable by reason of willful ignorance.
See any of the language shootouts: https://benchmarksgame-team.pages.debian.net/benchmarksgame/
Since you’re rewriting it anyway, redesign at the same time. By the third implementation I usually feel like the design is sorted. That holds pre-AI and now.
score 1 · canonical
until you have to do certain kinds of native interop, then the tracing system dies and so does the GC
score 1 · canonical
Are you talking from experience or speculating?
The best part about luajit is how good the interop is. Yes, like any JIT you have to watch out for things that trip it up. Same with GC. But, that’s not interop in my experience with luajit.
score 1 · canonical
Yeah. OpenMW ran into nasty edge cases around the tracing part of the JIT (and how it itself interacts with the GC), because of the specific way that the native interop is shaped. The people who maintain the lua integration part of the codebase are looking at luau because of it, which has something closer to a method JIT (I think it’s technically a JAOT?) and is specifically designed for game engines and seems to interact properly with the specific native interop shape that OpenMW needs.
If your native interop is normal then luajit’s way of supporting it is probably entirely unproblematic.
score 1 · canonical
Ah that’s interesting I’ll have to give that a look and see what that interop shape is as I haven’t run into myself. Thanks for the pointer.
Also absolutely love OpenMW!
score 1 · canonical
Python has entered the chat
This comment is meant to be flippant and meant to be a joke, but I think it’s worth hearing a bit from the cohort of Python users who are very performance sensitive—those of us who do data analysis or numerical computing.
Codes that rely on NumPy are often dismissively viewed as just “bolting” C onto Python, and the performance of the resulting system is, in my view, incorrectly attributed wholly to the capabilities of C. This really underestimates the amount of (evolved) “design” in these systems.
This is why I have maintained that the “restricted computation domain” formulation is useful when thinking about these systems. The idea that the “heavy lifting” or “number crunching” gets delegated to some C or Fortran codes (e.g., the aforementioned NumPy,) while the “programme structuring” or “business logic” happens within Python is an interesting but not particularly deep observation.
I don’t think there’s is much debate over whether Python semantics provide the ability to express information currently necessary to achieve performance on contemporary hardware—the deliberate choices (e.g., automatic integer promotion) and the missing pieces (e.g., control over memory layout) are quite well-documented. It remains to be seen whether these can be inferred or otherwise synthesised through code improvements (e.g., reducing the number of C-level stack frames required to perform a Python-level function call) and JIT (which, as I understand from talking to the core developers, has been successful in some benchmarks but has introduced some other undesirably complexity) and other approaches but, historically, the CPython interpreter just hasn’t been particularly fast at doing some fundamental operations.
Where the “restricted computation domain” formulation goes beyond simpler descriptions of Python as a glue language or observations that limitations in Python force heterogeneous computation (i.e., some in Python, some lower,) is that these entities require a significant amount of design to actually work. While a lot of the popular tools did not undergo a particularly extensive or deliberate design process, there were evolutionary factors at play that forced them to be mostly effective at encoding computations. However, it is actually not uncommon to “fall out” of the design of these tools.
For example, I spoke some years ago at PyData Amsterdam about “group-weighted summary statistics” (e.g., group-weighted standard deviation or z-transform with high cardinality of groups) using the popular pandas DataFrame library. From the most naïve approach using just pandas to a very contorted approach using just pandas to an approach relying on cython, I recall getting a 1× → 10× → 100× improvements. Similarly, when talking about developing simulators with Python and pandas, we must contend with computations that cannot readily be encoded into the tools themselves, and given the (polarised or dichotomous in nature) heterogeneity of these tools and the degree to which they cannot readily be decomposed, we end up having little choice but to accept the subsequent contortions.
All of this is to say… I genuinely simultaneously believe that:
- Python today is faster than Python was yesterday
- many fundamental operations may never be fast in Python
- naïve approaches (i.e., unaware of the designs discussed above) may inevitably lead to slow programmes
- the non-naïve approach does require design and is generally tenable without extraördinary effort
- a system that undergoes said design may be as fast or faster than an equivalent system in, say, C++ (and that other factors of C++ may contribute to this likelihood)
In fact, regarding the last point, a common refrain on a recent simulator project I worked on was that, if we had been involved in the original writing of that system, we could have written it in Python (and, of course, with C or Cython or C++ or Rust or Zig,) and it would have been easier to extend, debug, less code overall, and at least an order of magnitude faster if we had taken a naïve approach (and maybe even two or three orders of magnitude faster with a little sophistication…!!)
That means that I both agree and disagree with the thesis of this post. Python, as a language, doesn’t lend itself to being fast. Many Python programmes are not fast. Some or even many Python programmes are not fast, and gain nothing from using Python, where a line-for-line rewriting of them in another language would be faster. But the use of Python does not inevitably lead to systems that are slow, and, in fact, the necessary design to produce fast systems in Python may not (in terms of smarts and efforts and luck) be significantly more than what is necessary to achieve the same in, say, C++.
score 1 · canonical
Thank you for the elaborate reply, which my flippant comment didn’t deserve. I really appreciate your view of the system as a whole, where good system design is much more important than language choice.
However, an important point in my eyes is how many countless man-hours it takes to tend to the Python ecosystem, and ‘bend’ fast C/C++ libraries into working with Python. Python’s type system sucks, and you have to write a lot of glue to make it all work, and it takes time to maintain it because Python reinvents itself with every minor release (the 2to3-shift was also a great thing).
In a way everything is just ‘slapped on’, and I rarely see carefully designed python libraries. It’s like in the web development community, move fast or die. Only befitting that Python became the inofficial AI programming language.
score 1 · canonical
extraördinary
If you allow me the diversion, what accent of English do you speak? My understanding is that the diaeresis mark indicates that two consecutive vowels belong to different syllables, yet so far I have only heard extraordinary be pronounced with a single vowel sound in that position.
score 1 · canonical
This is a good point. I only know this from naïve (btw, the Icelandic keyboard layout is excellent, because it has an umlaut-modifier key, so ö, ä, ü, ï, ë, are all quickly typed). A few years ago I read that it’s permissible style to put that everywhere when two vowels touch, but it certainly is unorthodox.
score 1 · canonical
If you allow me the diversion, what accent of English do you speak?
I’ve given a few public presentations, so you can see what you can hear. (I used to be even more presentious and playful in the older ones, but I guess I’ve calmed down a bit…)
I know there has definitely been some drift recently on certain words, and I know that I was raised to speak a certain way (e.g., /njuːz/,) but I think some of that has worn off. But splitting the syllable in extraördinary doesn’t seem too bizarre?
score 1 · canonical
It appears to be regional. That’s the chiefly US pronunciation. Ext-raw-dinary is UK.
score 1 · canonical
Consider a simple CRUD website with a database. … In other words, most real programs are “I/O-bound”
Sigh. Another server-side developer who thinks all “real programs” are servers.
Outside of servers there are definitely some IO-bound programs (or parts of programs) but a lot that aren’t. A web browser’s layout engine, a 3d renderer, a word processor, a neural network…
score 2 · canonical
They said “most,” not all.
I think client-server application are ubiquitous and this is fair to say.
score 1 · canonical
In a client-server context, the sexy animation you play as a response to user interaction is not IO bound. Application start-up time is only partially IO bound. Very little about the client side of a client-server system is IO bound other than the specific case of waiting for packets from the server.
score 1 · canonical
Do you think that the statement: “most software is IO bound” contradicts: “some software is not IO bound?”
If so, we can talk about that. If not, I’m not sure what your point is with listing CPU bound tasks. Programs use CPU a lot. I don’t think there’s any disagreement there.
I feel like your examples are the prime examples of why most software is in fact IO bound. Typically, you don’t write the operating system, so you have a couple of levers for application start time, but aside from that this is the job of the OS.
You talk about animations, but I don’t think animations provide user value in most cases so aren’t super relevant.
Then you talk about waiting for packets from the server. Most applications that I can think of have value precisely because they store their data on a server and because that data is not trivial to store and retrieve. So, the app spends a lot of time waiting for that server.
score 1 · canonical
So, the app spends a lot of time waiting for that server.
This means the client is server-bound, not that the client (or server) is I/O bound. The server could be spending an unnecessary amount of time serializing things or processing the request slowly. It rarely reaches an I/O hardware bottleneck.
score 1 · canonical
Come on. When one process waits on another process, that’s called IO bound. This shouldn’t have to be said but I guess we’re arguing in bad faith now.
score 1 · canonical
I think this distinction is worth making, especially in the context of the post, which is arguing faster languages don’t make most software faster.
Certainly, if your client/server operation is bounded by the network connection, and the response time is comparable to ping time, no language or framework is going to make this faster. You’d need to either physically move the machines closer, or change the design.
But for client/server applications that do a lot of server-side computing, response time can dramatically exceed ping time, and rewriting the server in a faster language can absolutely get you a lot closer to the ping limit.
The first piece of software I ever put into production was an interface for querying SLURM clusters. The original Python script for doing the same thing took around 3s, and the Rust rewrite got this down to ~120ms. When I looked at the profiles, it turned out around ~70ms was spent waiting for the daemon to respond with the data, and no optimization, redesign, or language change would get me below that limit. But the rewrite moved us from being functionally process-bound to functionally I/O-bound.
So I’m very willing to believe that there’s a lot of software out there that is I/O-bound (in the sense of waiting for a response from a server) where a faster implementation of the server (or perhaps even the client) could dramatically speed up response time, even though ping hasn’t gotten faster, to the point of being I/O-bound (in the sense that you’re stuck physically waiting for packets to arrive).
score 1 · canonical
That’s a serialized communication dependency. The same category of “wait” occurs when a thread is blocked waiting on another, something I wouldn’t call “I/O”.
To me, “I/O” bound means the request perf/latency limited by the communication medium which you can’t control. Here, speeding up the server’s processing also lowers client latency, so the request isn’t bound by the client->server comms channel itself.
I don’t see how this is bad faith
score 1 · canonical
That’s a very uncommon definition of IO.
A network request has always been considered IO. Of course there is computation going on on the other end of the request that you equally have to optimize. But a client making frequent network requests and waiting on the result is still IO-bound.
score 1 · canonical
It’s also incorrect for server-side. The myth of the “I/O-bound” workload is strong in backend development unfortunately. Fetching data from DBs takes ~5ms but somehow average responses take ~500ms. There is a lot of CPU bound mini workloads and overhead people take for granted.
score 7 · canonical
Memory usage is an additional consideration. We expect a Go program to use an order of magnitude less memory than an analogous Python program. For demanding applications we may expect an optimized Rust program to use an order of magnitude less memory than the Go program. In server-side contexts this reduces the demand for horizontal scaling.
score 4 · canonical
Yes, great point with horizontal scaling. Which brings me to another thing I’d like to highlight: A Ruby/Python program will force you into design choices that a Rust/Go program will not because it can’t scale vertically.
At some point you’ll be forced to throw more machines at the problem which then means you’ll need: application load balancers, shared networked caching, shared networked file systems, networked inter-process communication in the form of API/gRPC/Kafka, and a whole lot more.
All of that adds latency. In a faster language a lot of that can be done in a single binary. So to escape a CPU/Memory bound problem we’ve incurred a lot of I/O-bound problems. Which is another reason why people think all server-side problems are I/O.
score 2 · canonical
What you describe definitely exists, with suitably chosen values for node CPU/RAM, but in many (most?) cases people try to draw that line at the weakest cloud node offering with some laughable amount of memory and are way too willing to go down the often over engineered horizontal scaling approach.
Like stackoverflow was famously using a single beefy machine a long time ago, I am fairly sure that doing the math properly would show that going 2 node classes higher (or to a dedicated machine) would end up cheaper in the end, compared to the development cost+minimal saving from up/down scaling. Like even just from an energy perspective it’s much more wasteful to have multiple machines doing a bunch of redundant work!
Oh and you can always just do FFI if needed.
score 1 · canonical
but in many (most?) cases people try to draw that line at the weakest cloud node offering with some laughable amount of memory and are way too willing to go down the often over engineered horizontal scaling approach.
At work I tend to point out that in Azure the SKUs scale linearly in the same family, i.e. one 4c/16GB costs exactly the same as two 2c/8GB instances during runtime.
This means that we shouldn’t really bother with horizontal scaling until 372c/1.4TB isn’t enough.
score 1 · canonical
It used to be a lot more correct for server-side than it is now, in the age of spinning disks. Circa 2006, Django requests were definitely usually dominated by database queries, and this was widely pointed out to people who complained about the speed of template formatting. That’s probably very rarely the case now.
score 2 · canonical
To try to put some concrete numbers on this:
Around 25 years ago, the server I deployed the little web apps I worked on to had a 1GHz CPU and a spinning-rust disk. The disk had a seek time of around 10ms and random disk reads topped out at about 150 KB/s. Linear reads were around 30 MB/s. Doing random disk accesses, I got a bit under 7M cycles per byte read from disk.
A modern server might have 32 3 GHz cores with NVMe storage. NVMe disks are ludicrously fast. Even a cheap one can manage over 100 MB/s in random reads, often closer to 300 MB/s and high-end ones are much faster. Assuming 300 MB/s, on a single core I get about ten cycles per byte for random reads. If I have a nicely multithreaded workload I get around 300 cycles per byte.
If my code 25 years ago had been I/O bound and sat with a CPU utilisation of 5%, the same code on a modern machine would peg a CPU core at 100% and I/O at 0.0003%. If I make it perfectly multithreaded, I/O usage becomes a massive 0.009%.
But I think this is focus on the specific case is hiding the real reason why software is slow: software where performance matters is heavily optimised but optimisation is a multi-dimensional constraint space and those constraints change over time. To give another example:
During my PhD, I was working with a codebase that had a load of globals for working state because repeatedly copying stack values was slow on the system it was originally written for. I refactored it to pass the state down as function arguments. On a more modern architecture, they were passed in registers and occasionally spilled. Globals required GOT-relative access and so were much slower. On top of that, making them local enabled the compiler to be less conservative about optimisation. The end result was that the code, now written in a more naïve style, was around 10-20% faster, with no algorithmic changes. This wasn’t because the original author was an idiot, it was because they were a clever person optimising based on constraints that no longer existed. The bottlenecks had moved.
The harder it is to refactor code or change underlying data structures, the more likely you are to be stuck with optimisation that was 100% necessary for performance goals 10-20 years ago but is now the cause of bottlenecks.
score 1 · canonical
The harder it is to refactor code or change underlying data structures, the more likely you are to be stuck with optimisation that was 100% necessary for performance goals 10-20 years ago but is now the cause of bottlenecks.
This is exactly why the previous product I worked on was such a nightmare. The original developers had had to do a lot of optimisations to do something genuinely impressive at the time, nowadays however that bit while not exactly trivial is still pretty straightforward. But since all the hacks were intermingled it was practically impossible to update to more modern standards.
score 1 · canonical
The harder it is to refactor code or change underlying data structures, the more likely you are to be stuck with optimisation that was 100% necessary for performance goals 10-20 years ago but is now the cause of bottlenecks.
I think this is getting at something general: even if optimization is local and not a property of major design decisions, having general good design makes it much easier to do these local optimizations. As another general phenomenon: often optimizations make code more complicated, and you need to offset that by having good control of complexity.
score 1 · canonical
Yeah, it’s interesting to me how we still largely design systems today to fit the design constraints of the late 90s.
Even relative to CPU speed, I/O speed has increased a lot, threads are orders of magnitude more performant, and memory access has gotten much more expensive.
score 5 · canonical
Even things like big-O analysis of algorithms that don’t do any I/O aren’t as simple as they were in the 90s. Nowadays an O(n^2) algorithm with a very CPU-cache-friendly memory access pattern can beat an O(n log n) one for surprisingly high values of n. In the 90s we didn’t have a 100x performance difference between memory reads depending on whether nearby memory has been recently accessed. (Though I guess if you squint, accessing swapped-out pages might sort of qualify.)
score 2 · canonical
And “O(1)” algorithms are actually roughly O(sqrt(n)), and “O(n)” algorithms are actually O(n sqrt(n)). As your data set grows, they spill out into larger, slower kinds of memory: L1 -> L2 -> L3 -> RAM -> local SSD -> local HDD -> NAS, giving a roughly sqrt(n) size/speed trade-off.
(I may be getting the actual function long, maybe it was log(n) and not sqrt(n), I got this from a talk I watched a while ago; but the idea is correct even if the specific function isn’t.)
score 1 · canonical
And “O(1)” algorithms are actually roughly O(sqrt(n))
One of the beefs I had with big-O when I was introduced to it in my CS classes in the 90s was that, as commonly practiced, it papers over some significant costs. Like a sorting algorithm that’s presented as O(n log n) only has that complexity if comparing two elements is a constant-time operation, which it isn’t if you’re, say, comparing arbitrary-length strings that can have common prefixes.
My college roommate and I invented a variant on radix sort for linked lists of variable-length values (by accident; neither of us had learned about any existing radix sort algorithms yet) and we were frustrated that using the analysis techniques we’d learned in class, it was an O(n) sorting algorithm, but the actual execution time clearly depended on the lengths of the values, not just the number of values.
I realize some of that is just “we give undergrads the simplified version” but I’d wager most people who learned big-O at all didn’t learn about the more sophisticated, comprehensive approach to it.
score 1 · canonical
Sounds like the presentation of Big-O didn’t explain it well. Big-O is only for comparing the relative count of one specific operation between two algorithms as the size of inputs changes - it was never intended to be a comprehensive performance metric.
In other words, the Big-O kind of factors out. For sorting, even if the comparisons aren’t constant time the number of non-constant-time comparisons still grows O(n log n) or O(n^2) or whatever.
score 1 · canonical