Kernel TCP Was 14× Slower Than Goblin Core’s Fastest Path

Adam DePrince — Mon, 27 Jul 2026 16:00:00 +0000

I am building low-latency AWS support for Goblin Core.

AWS EFA uses libfabric, but renting expensive cloud hardware is a bad time to discover that the client, server, completion queues, receive path, or small-message handling is broken. Before taking Goblin Core to a native EFA instance, I wanted to qualify the complete libfabric code path locally.

So I connected two 14-year-old, four-socket Intel Xeon systems with a direct 100 Gb/s Mellanox ConnectX-5 link and tested the combinations that mattered:

ordinary kernel TCP;
libfabric’s software tcp provider;
verbs;ofi_rxm over the Mellanox hardware;
Redis RESP2 and Goblin Core’s compact SBE protocol;
automatic small-message injection and forced fi_send.

Every workload used pipeline depth one: send one operation, wait for its complete response, then send the next. This was a latency test, not an attempt to fill a 100 Gb/s pipe. (Goblin Core)

The result

Across eight Redis-shaped operations, the fastest configuration completed the round trip in 6.39 microseconds at the arithmetic-average median.

The ordinary Redis-compatible baseline—RESP2 over kernel TCP—took 90.33 microseconds.

Path	Wire format	Average p50
Kernel TCP	RESP2	90.33 µs
Libfabric `tcp`	RESP2	39.59 µs
`verbs;ofi_rxm`	RESP2	7.38 µs
`verbs;ofi_rxm`	SBE	6.39 µs

That makes the fastest path 14.1 times lower latency than ordinary kernel TCP with RESP2. It also supports roughly 156,000 sequential round trips per second across the tested operations, compared with about 11,000 for the baseline. (Goblin Core)

The eight operations were:

PING, SET, GET, HSET, HGET, ZADD, ZSCORE, and Pub/Sub.

The Pub/Sub number includes more than acknowledging PUBLISH: timing ends only after a subscriber on a separate connection receives and validates the message. Each distribution contained 20,000 warmups followed by 200,000 measured operations, with every response checked for correctness. (Goblin Core)

Most of the win did not require abandoning RESP

The fastest result used SBE, Goblin Core’s compact binary protocol. But RESP2 over the same verbs;ofi_rxm transport still averaged 7.38 microseconds.

That is important.

The binary wire format helped, but the larger result came from changing the transport and software path. In this benchmark, a Redis-shaped RESP operation over the libfabric verbs path was more than twelve times faster than RESP2 over Goblin Core’s ordinary kernel TCP path. (Goblin Core)

The software tcp provider was also interesting. RESP2 over libfabric tcp averaged 39.59 microseconds—2.28 times lower than the ordinary socket implementation.

That does not establish a universal law that libfabric TCP is always faster than sockets. It says something narrower and more useful: inside this real server, the surrounding software path consumed a large fraction of the latency.

A 100 Gb/s card does not automatically give an application low latency. The protocol parser, buffer ownership, syscall pattern, polling model, completion handling, NUMA placement, and application architecture still decide how quickly one request becomes one validated response.

Then the tail latency went bad

The median result was only part of the work.

An earlier qualification run had an ugly, repeatable step in the tail. The average SBE/RDM p99.99 was roughly 857 microseconds—wildly out of proportion to a single-digit-microsecond median.

The obvious suspect was message reordering.

So I instrumented it.

Across nearly two million publisher replies and 220,001 subscriber replies, the client observed no missing sequence, duplicate, reorder-window overflow, stranded frame, or pending reordered message. Server request counters were clean too.

The transport was not losing control of the messages.

The stall came from a completely different place: Goblin Core’s bootstrap TCP listener.

The polled server called nonblocking accept() on every pass through its spin loop, whether or not a connection was waiting. On Linux, even an unsuccessful accept() may create and retire socket, file, and inode-related objects before returning EAGAIN.

Those objects eventually have to be reclaimed.

Tracing showed repeated RCU softirq batches, some lasting roughly 0.4 to 1.1 milliseconds. A listener intended only for bootstrap connections was periodically stalling the low-latency data path. (Goblin Core)

The fix was conceptually simple: check whether the listener is ready before calling accept(), and drain it only when a connection is actually waiting.

Before the fix, one trace contained 4,119 RCU softirq runs longer than 100 microseconds.

During a two-second active trace after the fix, every softirq run remained below 100 microseconds; the longest was 20.6 microseconds.

The immediate control run dropped average SBE/RDM p99.99 from 857.47 to 29.47 microseconds. The independent full provider matrix reproduced the improvement at 30.83 microseconds. (Goblin Core)

The boring code can own the latency

The listener was not part of the exciting design.

It was not RDMA. It was not the compact protocol. It was not the Redis data structure. It was not the 100 Gb/s NIC.

It was a small bootstrap path doing unnecessary work inside the wrong loop.

That is a recurring lesson in low-latency systems: the headline mechanism may be correct while something apparently harmless nearby destroys the tail.

The only reliable response is to measure the complete system, preserve anomalous results, and trace the mechanism instead of explaining the anomaly away.

What this result proves

It proves that Goblin Core’s libfabric RDM implementation works across two physical hosts with:

real request and response validation;
both RESP2 and SBE;
small-message injection;
explicit fi_send with transmit completion handling;
posted receives and receive completion queues;
Pub/Sub over separate publisher and subscriber connections;
bounded message reordering;
stable low-microsecond medians;
p99.99 behavior in the tens of microseconds after fixing the listener.

It also shows that the transport and surrounding software architecture can matter far more than the nominal speed of the network card.

What it does not prove

This is not yet an AWS EFA hardware benchmark.

The native efa provider was unavailable on these local machines. The measurements used AWS’s libfabric build with the tcp and verbs;ofi_rxm providers over a direct ConnectX-5 Ethernet link.

That distinction matters. EFA still needs its own run.

The purpose of this work was to reach AWS with a qualified implementation, a reproducible benchmark harness, validated send and receive paths, and known-good local baselines—not to rename a local verbs result as an EFA result. (Goblin Core)

Why this belongs on Goblin Reactor

This experiment is a small example of how I approach difficult systems work:

Establish the real measurement.

Test the complete matrix rather than the favorite configuration.

Keep the strange result instead of smoothing it away.

Instrument the suspected mechanism.

Change the implementation.

Run the entire experiment again.

The outcome was not simply a fast number. It was a validated network path, a repaired tail-latency failure, and a reproducible artifact ready for the next machine.

That is the work Goblin Reactor is built to do: take a hard systems problem through architecture, implementation, hostile measurement, and a result that survives contact with the machine.

The next stop is native AWS EFA.

What Goblin Core proves about performance work

Adam DePrince — Thu, 09 Jul 2026 16:00:00 +0000

Performance work should leave evidence behind. Goblin Core is one example of what that evidence can look like: not a vague claim that a system is “fast,” but measurements tied to a workload, a transport, and a machine.

The current headline numbers are:

51 bytes of RSS per sorted-set member at four million members.
14.3 billion real Lichess rating updates replayed.
220 nanoseconds for a single-element shared-memory request and reply.
392,000 ZADD operations per second on one core.

Those numbers describe different parts of the system. They should not be compressed into one universal speed claim.

Memory is a data-structure decision

At four million sorted-set members, Goblin Core uses 51 bytes of resident memory per member. That result comes from packed arenas and compact Swiss-table indexes instead of a graph of separately allocated objects.

The useful lesson is broader than Redis compatibility: when memory cost grows with millions of tiny objects, allocation strategy and data layout are often the product architecture. Treating them as implementation details can become very expensive.

Throughput needs a realistic history

The Lichess replay runs 14.3 billion real rating updates in chronological order and checks the result against Redis member for member. A long, real history exercises behavior that a small synthetic loop can miss: changing set sizes, repeat updates, ranking churn, and the cumulative effects of the storage design.

Latency includes the round trip

The 220 ns result is a synchronous, unpipelined request and reply for a one-field HSET or one-member ZADD over shared-memory SBE rings. The client and server are pinned to separate cores on a Threadripper PRO 5995WX, and the server busy-polls.

That context matters. The result shows what is possible when the kernel is removed from the hot path and a core is deliberately owned by the service. It is not a promise that an ordinary networked deployment will respond in 220 ns.

A benchmark should make the trade visible

The single-core ZADD result measures 392,000 operations per second. Together with the latency and memory numbers, it gives a more useful picture than any one result alone: how much work a core completes, how quickly one request returns, and how much memory the data consumes.

That is the standard Goblin Reactor brings to client work. First define the workload and the outcome that matters. Then measure the existing system, isolate the cost, build the smallest credible optimization, and state the tradeoffs alongside the result.

The full benchmark methodology and current results live at goblin-core.dev.

Goblin Reactor Field Notes