Kernel TCP Was 14× Slower Than Goblin Core’s Fastest Path

I am building low-latency AWS support for Goblin Core.

AWS EFA uses libfabric, but renting expensive cloud hardware is a bad time to discover that the client, server, completion queues, receive path, or small-message handling is broken. Before taking Goblin Core to a native EFA instance, I wanted to qualify the complete libfabric code path locally.

So I connected two 14-year-old, four-socket Intel Xeon systems with a direct 100 Gb/s Mellanox ConnectX-5 link and tested the combinations that mattered:

• ordinary kernel TCP;
• libfabric’s software tcp provider;
• verbs;ofi_rxm over the Mellanox hardware;
• Redis RESP2 and Goblin Core’s compact SBE protocol;
• automatic small-message injection and forced fi_send.

Every workload used pipeline depth one: send one operation, wait for its complete response, then send the next. This was a latency test, not an attempt to fill a 100 Gb/s pipe. (Goblin Core)

The result

Across eight Redis-shaped operations, the fastest configuration completed the round trip in 6.39 microseconds at the arithmetic-average median.

The ordinary Redis-compatible baseline—RESP2 over kernel TCP—took 90.33 microseconds.

Path	Wire format	Average p50
Kernel TCP	RESP2	90.33 µs
Libfabric tcp	RESP2	39.59 µs
verbs;ofi_rxm	RESP2	7.38 µs
verbs;ofi_rxm	SBE	6.39 µs

That makes the fastest path 14.1 times lower latency than ordinary kernel TCP with RESP2. It also supports roughly 156,000 sequential round trips per second across the tested operations, compared with about 11,000 for the baseline. (Goblin Core)

The eight operations were:

PING, SET, GET, HSET, HGET, ZADD, ZSCORE, and Pub/Sub.

The Pub/Sub number includes more than acknowledging PUBLISH: timing ends only after a subscriber on a separate connection receives and validates the message. Each distribution contained 20,000 warmups followed by 200,000 measured operations, with every response checked for correctness. (Goblin Core)

Most of the win did not require abandoning RESP

The fastest result used SBE, Goblin Core’s compact binary protocol. But RESP2 over the same verbs;ofi_rxm transport still averaged 7.38 microseconds.

That is important.

The binary wire format helped, but the larger result came from changing the transport and software path. In this benchmark, a Redis-shaped RESP operation over the libfabric verbs path was more than twelve times faster than RESP2 over Goblin Core’s ordinary kernel TCP path. (Goblin Core)

The software tcp provider was also interesting. RESP2 over libfabric tcp averaged 39.59 microseconds—2.28 times lower than the ordinary socket implementation.

That does not establish a universal law that libfabric TCP is always faster than sockets. It says something narrower and more useful: inside this real server, the surrounding software path consumed a large fraction of the latency.

A 100 Gb/s card does not automatically give an application low latency. The protocol parser, buffer ownership, syscall pattern, polling model, completion handling, NUMA placement, and application architecture still decide how quickly one request becomes one validated response.

Then the tail latency went bad

The median result was only part of the work.

An earlier qualification run had an ugly, repeatable step in the tail. The average SBE/RDM p99.99 was roughly 857 microseconds—wildly out of proportion to a single-digit-microsecond median.

The obvious suspect was message reordering.

So I instrumented it.

Across nearly two million publisher replies and 220,001 subscriber replies, the client observed no missing sequence, duplicate, reorder-window overflow, stranded frame, or pending reordered message. Server request counters were clean too.

The transport was not losing control of the messages.

The stall came from a completely different place: Goblin Core’s bootstrap TCP listener.

The polled server called nonblocking accept() on every pass through its spin loop, whether or not a connection was waiting. On Linux, even an unsuccessful accept() may create and retire socket, file, and inode-related objects before returning EAGAIN.

Those objects eventually have to be reclaimed.

Tracing showed repeated RCU softirq batches, some lasting roughly 0.4 to 1.1 milliseconds. A listener intended only for bootstrap connections was periodically stalling the low-latency data path. (Goblin Core)

The fix was conceptually simple: check whether the listener is ready before calling accept(), and drain it only when a connection is actually waiting.

Before the fix, one trace contained 4,119 RCU softirq runs longer than 100 microseconds.

During a two-second active trace after the fix, every softirq run remained below 100 microseconds; the longest was 20.6 microseconds.

The immediate control run dropped average SBE/RDM p99.99 from 857.47 to 29.47 microseconds. The independent full provider matrix reproduced the improvement at 30.83 microseconds. (Goblin Core)

The boring code can own the latency

The listener was not part of the exciting design.

It was not RDMA. It was not the compact protocol. It was not the Redis data structure. It was not the 100 Gb/s NIC.

It was a small bootstrap path doing unnecessary work inside the wrong loop.

That is a recurring lesson in low-latency systems: the headline mechanism may be correct while something apparently harmless nearby destroys the tail.

The only reliable response is to measure the complete system, preserve anomalous results, and trace the mechanism instead of explaining the anomaly away.

What this result proves

It proves that Goblin Core’s libfabric RDM implementation works across two physical hosts with:

• real request and response validation;
• both RESP2 and SBE;
• small-message injection;
• explicit fi_send with transmit completion handling;
• posted receives and receive completion queues;
• Pub/Sub over separate publisher and subscriber connections;
• bounded message reordering;
• stable low-microsecond medians;
• p99.99 behavior in the tens of microseconds after fixing the listener.

It also shows that the transport and surrounding software architecture can matter far more than the nominal speed of the network card.

What it does not prove

This is not yet an AWS EFA hardware benchmark.

The native efa provider was unavailable on these local machines. The measurements used AWS’s libfabric build with the tcp and verbs;ofi_rxm providers over a direct ConnectX-5 Ethernet link.

That distinction matters. EFA still needs its own run.

The purpose of this work was to reach AWS with a qualified implementation, a reproducible benchmark harness, validated send and receive paths, and known-good local baselines—not to rename a local verbs result as an EFA result. (Goblin Core)

Why this belongs on Goblin Reactor

This experiment is a small example of how I approach difficult systems work:

Establish the real measurement.

Test the complete matrix rather than the favorite configuration.

Keep the strange result instead of smoothing it away.

Instrument the suspected mechanism.

Change the implementation.

Run the entire experiment again.

The outcome was not simply a fast number. It was a validated network path, a repaired tail-latency failure, and a reproducible artifact ready for the next machine.

That is the work Goblin Reactor is built to do: take a hard systems problem through architecture, implementation, hostile measurement, and a result that survives contact with the machine.

The next stop is native AWS EFA.