This week I posted a video on YouTube and LinkedIn throwing some thoughts that were coming to my head week after week:
We all work with messaging systems: Kafka for high-throughput streams, AMQP for complex enterprise routing, MQTT for IoT and lightweight pub/sub. They're powerful, but most still run on raw TCP connections, just like they have for years.
Now, HTTP/3 (running on QUIC) is here, offering a ton of under-the-hood improvements: faster handshakes (0-RTT), built-in TLS 1.3+, no head-of-line blocking across streams, and connection migration (your client switches from Wi-Fi to mobile, the connection might actually survive). So, the question inevitably pops up: can we get our messaging workhorses to leverage this new transport without a total overhaul? And critically, can they still evolve independently?
Let me walk you through my thought process this week, including a few dead ends.
My first instinct was to think about HTTP/3's extensibility. "HTTP/3 has frames. Why not just create new ones? KAFKA_PRODUCE_FRAME, MQTT_PUBLISH_FRAME, AMQP_TRANSFER_FRAME. Direct. Clean."
Sounds logical, right? On the surface, it’s a direct mapping.
Then you dig a bit deeper. HTTP/3 frames (DATA, HEADERS, SETTINGS, PRIORITY_UPDATE, etc.) are the nuts and bolts for how HTTP itself functions – managing requests, responses, control information, and features of the HTTP protocol.
If we started injecting KAFKA_PRODUCE_FRAME, we wouldn't really be speaking HTTP/3 anymore. We'd have a custom protocol that looks a bit like HTTP/3. Standard HTTP/3 clients, servers, proxies, and debug tools? They wouldn't have a clue what to do with it. We'd also lose all the semantic value of HTTP methods (GET, POST), status codes, and standard headers. Plus, trying to standardize such application-specific frames across the IETF would be… ambitious, to put it mildly 😅
So, custom frames are out for this purpose.
Then I thought: alright, if custom frames are a no-go, what about using actual HTTP requests?
- Kafka
Produce -> POST /topics/my-topic/messages
- MQTT
Publish -> POST /mqtt/topics/device-updates
- AMQP
Transfer -> POST /amqp/exchanges/my-exchange/publish
This is much more HTTP-idiomatic. You get to use existing HTTP libraries, and conceptually, it's clear. For many request/response style interactions in these protocols, this could work.
My brain started screaming "HACK! DON'T FOLLOW THIS PATH!" 😄
It's definitely not a perfect fit. Not even a good one, at least not for a protocol. The persistent, stateful, bidirectional nature of many messaging sessions (think AMQP channels or an MQTT client constantly subscribed and receiving messages) can feel clunky when forced into a pure HTTP request/response model for every interaction. And how do you ensure each protocol can still add its own unique, non-RESTful features down the line without being constrained by HTTP methods? Nah, discarded!
Then I remembered my old days at the university, coding the network protocols. What about tunneling? This brings us closer. Remember how WebSockets "upgrade" an HTTP/1.1 connection? HTTP/3 has a similar mechanism called "Extended CONNECT" (defined in RFC 9220 for bootstrapping WebSockets, but the concept is general).
The idea:
- Client makes an HTTP/3
CONNECT request.
- Includes a header indicating the desired protocol, e.g.,
protocol: kafka or Sec-Kafka-Protocol: v2.8.
- If the server agrees, the HTTP/3 stream (or potentially the whole connection) effectively becomes a raw binary tunnel for the specified protocol.
This is pretty good. It leverages HTTP/3 for the initial secure, multiplexed handshake and then gets out of the way, letting the native protocol flow. This definitely allows for independent evolution.
While Extended CONNECT is solid for tunneling, there's another IETF/W3C effort that feels even more tailor-made: WebTransport.
I found WebTransport completely by chance searching on Google. It is built on top of HTTP/3 and is specifically designed for client-server messaging that needs more than traditional HTTP request/response. Think of it as a framework for diverse application protocols to run over HTTP/3. Looks like I was not the only one thinking about this possibility! 🎉
Here’s the flow:
- Client establishes an HTTP/3 connection to a server.
- Client sends a specific HTTP/3
CONNECT request to a designated WebTransport endpoint (e.g., https://messaging.example.com/.well-known/kafka-service-wt). This request itself can signal the intent to speak Kafka, AMQP, etc., perhaps by different endpoint paths or a custom header.
-
Once the WebTransport session is established, the magic happens:
- It provides an API to open multiple bidirectional and unidirectional QUIC streams.
- It also supports sending unreliable QUIC datagrams.
- Over these streams (or datagrams), the client and server exchange the native binary messages of Kafka, AMQP, MQTT, or whatever protocol was negotiated for that session.
Why I think WebTransport Hits the Sweet Spot:
- Native Protocol Preservation: Kafka still speaks Kafka binary. MQTT still speaks MQTT binary. No re-mapping of operations to HTTP methods if you don't want to. This is huge for "independent evolution."
- Full QUIC Benefits: You get the multiplexing, security, and connection management of QUIC/HTTP/3.
- Flexibility: Reliable streams map perfectly to most messaging operations. Unreliable datagrams could be a fit for things like MQTT QoS 0 or non-critical telemetry.
- Standardized Approach: It's being developed as a standard, aiming to be a common way to run such applications over HTTP/3.
So, a Kafka client could initiate a WebTransport session, tell the server "let's talk Kafka," and then just start firing off its usual binary ProduceRequests and WorkspaceRequests over WebTransport streams. Same for an MQTT client with its CONNECT, PUBLISH, SUBSCRIBE packets.
Obviously, this would still require message brokers to accept connections via WebTransport. Or event gateways like Gravitee, Hoockdeck, and others to support it. It's not an easy task but it's definitely easier than standardizing new HTTP/3 frames for messaging protocols 😅
If we want to bring our existing, powerful messaging protocols into the HTTP/3/QUIC era without rewriting their cores or boxing them into overly rigid HTTP semantics, WebTransport looks like the most promising path. It offers a standardized session layer that respects the integrity and independent evolution of the protocols running on top of it.
It's still relatively early days for WebTransport adoption compared to raw TCP or even WebSockets, but the foundation is solid and designed for exactly this kind of challenge. Definitely something to keep an eye on (or even experiment with) as we look to modernize our infrastructure. And a good place for people looking to contribute to open source and standards!
When I first published the video, I wasn't even aware of WebTransport existence. I'm happy they started this effort. It's now technically possible to communicate with a server using a binary protocol. In other words, it opens the doors for my dreamed capability: interacting with a broker from the browser. I'm quite excited this is now "possible".
From time to time, I love to challenge the status quo —even if just in my head. This lets me visualize where each piece of the puzzle fits in. It's always (always!) a good idea to understand the reasons why things are the way they are today. But you gotta dream. Dreams are free and sometimes they ignite the flame of future interesting initiatives, if you know what I mean 😉
