lobsters score: 30 · tags: networking
I wonder how many systems communicate via HTTP that would see a non-trivial performance increase if they implemented a custom protocol. I don’t think it would really change much, since HTTP overhead isn’t going to be what makes the difference in the number of packets you have to send for something. So maybe this is a good thing, because having a standard, even if it’s a standard that was originally developed for hypertext documents, is still worth something.
score 9 · canonical
Over the course of my career, I’ve come across something like O(100) custom protocols for service-to-service communication, all built with the assumption that e.g. JSON-over-HTTP would be too inefficient. These protocols were almost always underspecified, fragile, and fiendishly difficult to maintain. (No shade to their authors on these points – protocol design is hard!)
At some point, I started applying a test. I would write an end-to-end benchmark for the system as a whole, i.e. not a micro-benchmark of an individual parser or component. I’d run the benchmark with the default custom protocol to capture baseline results. I’d then write an alternative protocol with bog standard HTTP clients and servers, sending and receiving simple JSON objects, using gzip compression. And in almost all cases, gzipped JSON-over-HTTP would exhibit both lower latency and higher throughput than the custom protocol.
I’m not saying this is an absolute truth. There are definitely exceptions. But those exceptions gotta be justified with tests and benchmarks. In short, I agree with you :)
score 35 · canonical
These protocols were almost always underspecified, fragile, and fiendishly difficult to maintain.
… And invariably require special tooling to debug. Often said tooling either doesn’t get written, or is ad hoc and thrown together at the last minute out of necessity. There is a lot to be said for a protocol that you can just read and write when you’re in the development phase.
score 6 · canonical
And invariably require special tooling to debug.
It should be a good practice that each author of such a protocol or format also creates a dissector for Wireshark and some standalone tool and library for parsing and generating.
score 2 · canonical
Pieter Hintjens alluded to this pattern in his Cheap and Nasty protocol design article: you want either something easy and simple, or brutally tuned for sheer throughput. Trying to mix them or going straight to Nasty without a good reason is a recipe for pain.
score 4 · canonical
I imagine these days using zstd instead of gzip would widen the gap even further.
score 2 · canonical
It depends on the specifics of the payload, but in general, it’s less impactful than you might think. The difference to gzip is typically O(1-10%), which is usually lost in the noise, especially if you’re using HTTP/2. (Like you should be!)
score 1 · canonical
if you’re using HTTP/2. (Like you should be!)
Why? I’ve seen no difference between http 1.1 and 2 in practice on any of my cases.
score 1 · canonical
Basically, it’s about request multiplexing over connections.
HTTP/1 connections serve one request at a time, which means N concurrent requests require N active connections. But if every active request to a server requires a unique connection, then connection overhead (in the broadest sense) quickly becomes the bottleneck in the system. What you want instead is to mux arbitrarily many logical requests over a single physical connection. This is what HTTP/2 gives you. Makes a huge difference in high-RPS systems.
score 1 · canonical
HTTP/2 is an improvement, but muxing over a single tcp connection is still problematic as HTTP/2 still exhibits head-of-the-line blocking at the TCP layer – I imagine in really high RPS systems it would still be suboptimal. HTTP/3 should provide a big improvement though in that regard.
score 1 · canonical
Yep! Although it’s worth observing that fixing request-per-connection (by moving from HTTP/1 to HTTP/2) is going to deliver performance benefits that are something like an order of magnitude greater than the benefits from fixing TCP head-of-line blocking (by moving from HTTP/2 to HTTP/3) in the nominal case. Both are worth doing! It’s just the classic situation of diminishing returns.
score 1 · canonical
O(100) = O(1), fyi
(The set of functions which are asymptotically bounded by a straight line of any gradient.)
score 7 · canonical
You’re certainly correct.
On the other hand, O(x) also colloquially means “on the order of”, to estimate a quantity to some number as a lowest upper bound. I suppose writing 10 < x < 100 would be more accurate, or x+ = 100, or some other notation that expresses bounds. Obviously he is not talking about the complexity of an algorithm here, so we all understood what he was saying even if he “misused” the notation.
score 2 · canonical
While I don’t disagree with the general message here, this sentence stuck in my throat:
“The conceptual model of HTTP is also simple, in that you can basically view it as an RPC system.”
Ugh. That’s exactly what HTTP is not, conceptually or otherwise.
score 27 · canonical
In what way is HTTP — or, at least, HTTP POST — not RPC?
REST certainly isn’t RPC, sure, but REST is an abstraction layer above HTTP.
score 6 · canonical
In that HTTP is all about manipulating a large set of resources (nouns) with a very small set of fixed commands (verbs), and there are no remote procedures at a conceptual level at all.
score 1 · canonical
Right. As I mentioned in a sibling comment, I think I’m using a weaker definition of RPC than what you’re describing here. For me — and I think for most developers in this day and age — RPC is anything which is synchronous and follows request-response semantics. By this definition, all HTTP operations qualify. I understand this is not the original meaning.
score 2 · canonical
The words are all messed up at this point, but RPC traditionally meant mechanisms like “procedure naming” and “marshal arguments and return values”. It’s designed to make remote procedure calls as convenient as local ones with respect to particular programming languages.
So I’d say HTTP isn’t RPC because it’s not designed for, or convenient for, that purpose.
You can make it do such things, but it takes extra work, and you also lose things like HTTP proxy support (middleboxes).
If HTTP were RPC, then you wouldn’t have additional protocols like JSON-RPC tunneled inside HTTP. JSON-RPC specifies how your arguments and return values are marshaled.
If HTTP were RPC, you wouldn’t have to say “HTTP POST”. Middleboxes understand HTTP verbs; they often don’t understand your custom protocol inside HTTP. The network isn’t just about the end points.
Likewise, TCP/IP isn’t a protocol for hypertext, although you can use it that way. You have another layer with more semantics.
score 38 · canonical
I get the reluctance but I still think it’s somewhat semantic from the server’s perspective:
- HTTP verbs are cache controls, which is not unusual for RPCs (otherwise would be done in request metadata/headers/etc)-- middleboxes are generally aligned with this.
- HTTP endpoints (or “resources”) are encoded procedure names
To me, HTTP is a somewhat specialized kind of RPC (with specified caching and method encoding). We can of course build other kinds of RPCs on top of any RPC too.
We can draw the line elsewhere, but at the very least HTTP is very close to being an RPC. :)
score 2 · canonical
I suppose the idea of “HTTP is RPC” takes a minimalist definition of both terms-- RPC as in “it invokes procedures remotely”, and HTTP as in “a well-supported presentation layer”.
score 2 · canonical
Ah, okay, I’m definitely using a weaker definition of RPC than what you’re describing here.
score 7 · canonical
I mean Whois is definitely an RPC system: it has a verb! Sure, it’s implicit, but still!
score 4 · canonical
I don’t want to say that HTTP is a good RPC system or that it has all of the features of a proper one, but at a high level I think what a lot of people want out of their RPC is ‘make fixed call, get results and/or status’. HTTP requests with JSON payloads work pretty well for this and they don’t require, say, a connection setup phase the way a SMTP-like or IMAP-like protocol might have. At a low level HTTP can have keepalives and connection reuse and so on, but I think that tends to be either hidden away in the HTTP library or optional.
(I’m the author of the original, linked-to entry.)
score 8 · canonical
You don’t have to use HTTP
I don’t know, I feel like firewalls increasingly make sure I do.
score 7 · canonical
This 15-years-ago talk by Leonard Richardson provides useful perspective:
What happened? There are fewer big-name Internet protocols now than in 1993. Well, the Web ate those other protocols, just like the Internet ate Compuserve and BITNet and the UUCPNET and a bunch of other networks you’ve never heard of. It turns out most of the things we want to do on the Internet–have discussions, search databases, share files, look up things in directories–we can do over the web.
— https://www.crummy.com/writing/speaking/2008-QCon/
The whole thing is excellent and a useful bit of Silicon Valley / Cold War history.
score 6 · canonical
Also see
- HTTP as the Narrow Waist of the Future Internet (Berkeley 2010) – I think it was already the narrow waist by 2010, but people still thought of it as a “hack”. It’s better to understand the reasons why, and explicitly frame it that way, which will lead to better design and engineering
We then investigate HTTP’s limitations, and propose an extension, called S-GET that provides support for low-latency applications, such as VoIP and chat.
Looking back on this from over a decade later, this might have been basically what Web Sockets and WebRTC are?
and
We argue that HTTP is highly evolvable, due to (i) naming flexibility, (ii) indirection support, and (iii) explicit middleboxes. We point to evolving uses of HTTP on today’s Internet, and designing our own publisher/subscribe service, HTTP Relay Service (HTTP-RS), on top of HTTP.
More
- https://book.systemsapproach.org/e2e/trend.html
- My own post: The Internet Was Designed With a Narrow Waist – lots of history in the appendix, especially
score 6 · canonical