Half a Million RPS on a GB10: The Network Was the Bottleneck, Not the Box

We build our cloud on a single Go binary — one artifact that mounts the AI gateway, commerce, IAM, and KMS and serves api.hanzo.ai. Its HTTP surface is zip, our ZAP-native web framework, wrapped over fasthttp. So we asked a simple question: how fast can this thing actually go, and what stops it?

The honest answer surprised us three times. Here's the whole detective story, numbers and all. Everything is public in hanzoai/benchmarks.

First: is the framework the tax?

No. zip carries a constant ~24 nanoseconds and a single 48-byte allocation over the raw fiber/fasthttp engine it wraps — under 1% on any real handler. End to end, on loopback, zip does 0.95× of raw fasthttp. The framework is noise. Good. Move on.

framework	req/sec
raw fasthttp	365–400k
zip (HTTP)	338–348k

Second: the distributed result got slower

Here's where it got interesting. We have two boxes on a direct 2.5 GbE wired link: spark (an NVIDIA GB10, 20 cores) as the server, evo (32-core x86) as the load generator. Moving the load off-box should free spark's cores to serve — we expected it to go up.

It went down. 194k req/sec. And when we added connections to push harder, it got worse: 167k at 4,000 connections, 154k at 8,000, with p99 latency climbing from 5 ms to 52 ms. That's textbook congestion collapse.

But the link was only at ~70 MB/s — 22% of 2.5 GbE. Not bandwidth. And spark's 20 cores were mostly idle. Not CPU. So what?

The culprit: one receive queue

ethtool -l gave it away: the NIC was running with a single hardware RX queue. Every inbound packet was being processed by one core's softirq — and that one core was the ceiling. Worse, the driver flatly refused to add queues (ethtool -L → "Operation not supported").

So the fastest developer box we own was being throttled by a single kernel thread shoveling packets, while nineteen cores watched.

The fix: steer packets in software

When the hardware won't multi-queue, the kernel will — Receive Packet Steering hashes flows across cores in software. We turned on RPS (and RFS/XPS) across all 20 cores, raised the receive backlog from 1,000 to 300,000, set 256 MiB socket buffers, and pinned the performance governor. One idempotent tune.sh, run on both boxes.

Same wire. Same handler. 518,000 req/sec — a 2.67× jump, peaking at 696k.

concurrency	untuned	tuned
1,000	194k	518k
4,000	154k	474k (peak 696k)

The kicker: the tuned network number (518k) now beats loopback (365–400k) — because over the wire spark's cores serve full-time instead of sharing with the load generator. At 518k the box idles only ~12%. That's close to the real ceiling from a single loader.

Third: ZAP was supposed to be faster — so we made it

zip is ZAP-native — a bare listen address speaks our binary transport; http:// speaks HTTP. We expected the binary transport to win, built a ZAP load client (HTTP tools can't speak it), and measured.

ZAP did 91k. HTTP did 481k. The native transport lost — by 5×.

We chased it, and it's not the wire — it's the codec. The latency tail told the story (p50 a healthy 303 µs, but p99 at 13 ms — a garbage-collection signature), and relaxing GOGC from 100 to 800 more than doubled ZAP to 203k. Over a six-second run the ZAP server triggered 2,203 garbage collections; the HTTP server, 57. Three allocation sites carried it: the marshaler mints a fresh frame buffer per call, header decode copies every string into a map, and — the big one — zaphttp's server allocated a fresh request context per request where fasthttp's own server pools it per connection.

So we fixed it. The change (zap-proto/http, branch perf/zero-alloc-codec) is mechanical, not architectural: pool the request context once per connection, hand-roll a frame encoder that appends into a pooled buffer, and drain the 4-byte length prefix through bufio.Peek/Discard so no header array escapes to the heap. The serve loop serializes with AppendResponse into that reused buffer and decodes headers in place — so over a warm connection it allocates nothing per request — and golden tests confirm the wire is byte-for-byte identical before and after. Pure codec rewrite, no protocol change.

The result: ZAP goes 91k → 570k req/sec, a 6.3× lift that overtakes HTTP's 481k — ZAP now wins by 1.18× on loopback, with p99 collapsing from 13 ms to 1.19 ms and GC falling from 2,203 to 62.

Two things I had to correct myself on — and both survive the fix, so they're worth stating plainly. First, the win is CPU, not the wire. The ZAP frame is actually larger than HTTP (124 vs. 75 bytes for a /health request); ZAP wins by spending fewer cycles per byte, not fewer bytes. Second, zaphttp does not skip JSON. It carries the HTTP body as opaque bytes on both transports, so a handler pays the same marshal cost either way — on a chat-shaped JSON payload ZAP is in fact 0.95× HTTP, a slight loss from copying the body into the frame tail. JSON-elision is a property of native ZAP typed RPC, a separate axis zaphttp doesn't exercise. The loopback win here is honest and purely a CPU/allocation win. And over the real 2.5 GbE wire? Both transports tie at ~530k — the link is saturated and the codec is no longer the bottleneck.

Fourth: so does ZAP actually skip JSON?

That last line — "JSON-elision is a native-ZAP property zaphttp doesn't exercise" — is a promissory note, and a reader should make me cash it. zaphttp carries the body opaque, so it never tests the thing binary protocols are supposed to win at: reading structured data without parsing text. So I built the test that does.

One shared record — an eth_getBalance-shaped call, four fields each way (account, block, id → balance, nonce, blockHash) — round-tripped four ways: stdlib encoding/json, goccy/go-json and sonic (the two fastest JSON libraries in Go), and zap-proto/go's typed binary. Same fields in, same compute, same fields out; a round-trip-equivalence test enforces byte-exact re-encoding so neither side can skip work the other does. And I held JSON to its fastest implementation, not stdlib, so nobody can wave the result off.

Decode is a massacre. Reading the four-field request:

codec	decode	allocs
JSON stdlib	2334 ns	13
JSON sonic	845 ns	8
JSON goccy	473 ns	4
ZAP typed	13 ns	0

Thirteen nanoseconds, zero allocations — 35× faster than the fastest JSON, 174× faster than stdlib. There's no parse: the field sits at a known offset, you read it. That's the entire pitch of a typed binary wire, and it holds up.

But I promised to report the parts that don't flatter ZAP, so:

• Encode is a wash, and it allocates more. ZAP serializes in 629 ns with 12 allocations vs. goccy's 426 ns / 4. The published zap-proto/go v1.3.0 builder defers variable-length tails through a copy and heap-escapes its object builder; the pooled fixed-width API that fixes it isn't in the released tag, so that's the honest number. Netted out, the full server handler (decode + compute + encode) does favor ZAP — 274 ns vs. goccy's 431 — because the decode win swamps the encode wash. But it's decode carrying it.
• End-to-end, the 174× shrinks to ~1.2×. Over a real /rpc endpoint: ZAP 535k req/s, sonic 483k (1.11×), goccy 452k (1.18×), stdlib 407k (1.31×). Real, but nowhere near the microbenchmark gap — because at half a million requests a second the socket and the scheduler are the cost, and serialization is a thin slice of it. The codec stops being the bottleneck long before you'd notice it got faster.

One surprise that cuts for ZAP: on a real structured record its wire is smaller, not larger — 162 vs. 233 bytes — because JSON spends bytes on 0x-hex strings, decimal integers, quotes, and field names. The "ZAP is bigger" caveat from the framing test only holds for trivial /health-sized payloads; give it real typed fields and binary wins on size too.

So, honestly: does ZAP skip JSON? Yes — and on the read path it isn't close. Whether you feel it depends on whether serialization is your bottleneck. At 500k rps of tiny records it isn't. Swap in fat nested payloads or a decode-heavy fan-out and that 174× starts to show up where you live.

What we learned

• Benchmark the whole path, and question the tool. hey capped us at ~100k — that was the client, not the server. bombardier revealed 4× more.
• The box is rarely the first bottleneck. A single NIC RX queue cost us 2.67× before we touched a line of application code.
• Measure allocations, not just throughput. ZAP "being slow" was one GOGC flag away from being obviously a codec problem — and once we pooled the context and zeroed the allocations, the native transport overtook HTTP.
• A microbenchmark and a load test answer different questions. Typed decode is 174× faster than JSON in isolation and 1.2× faster end-to-end — both numbers are true, and quoting only one of them would be a lie of omission.
• Report it fair and real. Same handler, warm connections, no cherry-picked GOGC, JSON held to its fastest library not its slowest, publish the script. When the honest number was a loss — the encode wash, the compressed end-to-end margin — we shipped the loss too.

The zero-allocation ZAP codec is done, and it wins. The remaining stops are physical: a dual-NIC loader (parallel receive paths toward seven figures), faster interconnect (10 GbE / RDMA), a writev split so ZAP's frame tail skips one body copy on large payloads, and the post-quantum transports Go 1.26 gives us natively — PQ-TLS 1.3, ZAP-over-PQ-TLS, and PQ-QUIC — where we'll separate the ML-KEM handshake cost from steady-state throughput.

We can take this as far as we need to.