I wonder how many systems communicate via HTTP that would see a non-trivial performance increase if they implemented a custom protocol. I don’t think it would really change much, since HTTP overhead isn’t going to be what makes the difference in the number of packets you have to send for something. So maybe this is a good thing, because having a standard, even if it’s a standard that was originally developed for hypertext documents, is still worth something.
Over the course of my career, I’ve come across something like O(100) custom protocols for service-to-service communication, all built with the assumption that e.g. JSON-over-HTTP would be too inefficient. These protocols were almost always underspecified, fragile, and fiendishly difficult to maintain. (No shade to their authors on these points – protocol design is hard!)
At some point, I started applying a test. I would write an end-to-end benchmark for the system as a whole, i.e. not a micro-benchmark of an individual parser or component. I’d run the benchmark with the default custom protocol to capture baseline results. I’d then write an alternative protocol with bog standard HTTP clients and servers, sending and receiving simple JSON objects, using gzip compression. And in almost all cases, gzipped JSON-over-HTTP would exhibit both lower latency and higher throughput than the custom protocol.
I’m not saying this is an absolute truth. There are definitely exceptions. But those exceptions gotta be justified with tests and benchmarks. In short, I agree with you :)
These protocols were almost always underspecified, fragile, and fiendishly difficult to maintain.
… And invariably require special tooling to debug. Often said tooling either doesn’t get written, or is ad hoc and thrown together at the last minute out of necessity. There is a lot to be said for a protocol that you can just read and write when you’re in the development phase.
It should be a good practice that each author of such a protocol or format also creates a dissector for Wireshark and some standalone tool and library for parsing and generating.
On the other hand, O(x) also colloquially means “on the order of”, to estimate a quantity to some number as a lowest upper bound. I suppose writing 10 < x < 100 would be more accurate, or x+ = 100, or some other notation that expresses bounds. Obviously he is not talking about the complexity of an algorithm here, so we all understood what he was saying even if he “misused” the notation.
It depends on the specifics of the payload, but in general, it’s less impactful than you might think. The difference to gzip is typically O(1-10%), which is usually lost in the noise, especially if you’re using HTTP/2. (Like you should be!)
Basically, it’s about request multiplexing over connections.
HTTP/1 connections serve one request at a time, which means N concurrent requests require N active connections. But if every active request to a server requires a unique connection, then connection overhead (in the broadest sense) quickly becomes the bottleneck in the system. What you want instead is to mux arbitrarily many logical requests over a single physical connection. This is what HTTP/2 gives you. Makes a huge difference in high-RPS systems.
HTTP/2 is an improvement, but muxing over a single tcp connection is still problematic as HTTP/2 still exhibits head-of-the-line blocking at the TCP layer – I imagine in really high RPS systems it would still be suboptimal. HTTP/3 should provide a big improvement though in that regard.
Yep! Although it’s worth observing that fixing request-per-connection (by moving from HTTP/1 to HTTP/2) is going to deliver performance benefits that are something like an order of magnitude greater than the benefits from fixing TCP head-of-line blocking (by moving from HTTP/2 to HTTP/3) in the nominal case. Both are worth doing! It’s just the classic situation of diminishing returns.
Pieter Hintjens alluded to this pattern in his Cheap and Nasty protocol design article: you want either something easy and simple, or brutally tuned for sheer throughput. Trying to mix them or going straight to Nasty without a good reason is a recipe for pain.
I wonder how many systems communicate via HTTP that would see a non-trivial performance increase if they implemented a custom protocol. I don’t think it would really change much, since HTTP overhead isn’t going to be what makes the difference in the number of packets you have to send for something. So maybe this is a good thing, because having a standard, even if it’s a standard that was originally developed for hypertext documents, is still worth something.
score 9 · canonical
Over the course of my career, I’ve come across something like O(100) custom protocols for service-to-service communication, all built with the assumption that e.g. JSON-over-HTTP would be too inefficient. These protocols were almost always underspecified, fragile, and fiendishly difficult to maintain. (No shade to their authors on these points – protocol design is hard!)
At some point, I started applying a test. I would write an end-to-end benchmark for the system as a whole, i.e. not a micro-benchmark of an individual parser or component. I’d run the benchmark with the default custom protocol to capture baseline results. I’d then write an alternative protocol with bog standard HTTP clients and servers, sending and receiving simple JSON objects, using gzip compression. And in almost all cases, gzipped JSON-over-HTTP would exhibit both lower latency and higher throughput than the custom protocol.
I’m not saying this is an absolute truth. There are definitely exceptions. But those exceptions gotta be justified with tests and benchmarks. In short, I agree with you :)
score 35 · canonical
… And invariably require special tooling to debug. Often said tooling either doesn’t get written, or is ad hoc and thrown together at the last minute out of necessity. There is a lot to be said for a protocol that you can just read and write when you’re in the development phase.
score 6 · canonical
It should be a good practice that each author of such a protocol or format also creates a dissector for Wireshark and some standalone tool and library for parsing and generating.
score 2 · canonical
O(100) = O(1), fyi
(The set of functions which are asymptotically bounded by a straight line of any gradient.)
score 7 · canonical
s/something like O(100)/on the order of 100/score 3 · canonical
You’re certainly correct.
On the other hand, O(x) also colloquially means “on the order of”, to estimate a quantity to some number as a lowest upper bound. I suppose writing 10 < x < 100 would be more accurate, or x+ = 100, or some other notation that expresses bounds. Obviously he is not talking about the complexity of an algorithm here, so we all understood what he was saying even if he “misused” the notation.
score 2 · canonical
I imagine these days using zstd instead of gzip would widen the gap even further.
score 2 · canonical
It depends on the specifics of the payload, but in general, it’s less impactful than you might think. The difference to gzip is typically O(1-10%), which is usually lost in the noise, especially if you’re using HTTP/2. (Like you should be!)
score 1 · canonical
Why? I’ve seen no difference between http 1.1 and 2 in practice on any of my cases.
score 1 · canonical
Basically, it’s about request multiplexing over connections.
HTTP/1 connections serve one request at a time, which means N concurrent requests require N active connections. But if every active request to a server requires a unique connection, then connection overhead (in the broadest sense) quickly becomes the bottleneck in the system. What you want instead is to mux arbitrarily many logical requests over a single physical connection. This is what HTTP/2 gives you. Makes a huge difference in high-RPS systems.
score 1 · canonical
HTTP/2 is an improvement, but muxing over a single tcp connection is still problematic as HTTP/2 still exhibits head-of-the-line blocking at the TCP layer – I imagine in really high RPS systems it would still be suboptimal. HTTP/3 should provide a big improvement though in that regard.
score 1 · canonical
Yep! Although it’s worth observing that fixing request-per-connection (by moving from HTTP/1 to HTTP/2) is going to deliver performance benefits that are something like an order of magnitude greater than the benefits from fixing TCP head-of-line blocking (by moving from HTTP/2 to HTTP/3) in the nominal case. Both are worth doing! It’s just the classic situation of diminishing returns.
score 1 · canonical
You presume that the other protocols have no compression?
score 1 · canonical
Pieter Hintjens alluded to this pattern in his Cheap and Nasty protocol design article: you want either something easy and simple, or brutally tuned for sheer throughput. Trying to mix them or going straight to Nasty without a good reason is a recipe for pain.
score 4 · canonical