Connex

Endpoint mobility for dynamic LLM serving. Join, leave, fail, and migrate with traffic still in flight.

Connex. Endpoint mobility primitives for dynamic LLM serving.

Connex is open. It is the system in Connex: Endpoint Mobility Primitives for Dynamic LLM Serving (SIGCOMM 2026) (Lin et al., 2026). A serving endpoint keeps one logical name while the process behind it joins, leaves, fails, or moves, and steps that were already in flight stay correct across that change. The code is at HeterInfra/connex.

Collective libraries fix the set of peers at startup. A rank is an address, and a lost rank ends the job. Serving does not stay in that shape. Replicas come and go with load. A pipeline stage dies and is replaced. One request is often served by several role-specific processes. The peer of a running request can change while its tensors are still on the wire. Connex treats that as a delivery problem, and gives the application a mobile endpoint.

Connex control plane beside a pluggable data plane. connex-rs is the default when RDMA is present.
Connex owns who talks to whom. A pluggable data plane moves the bytes. On a host with RDMA, the default is connex-rs.

The control plane keeps logical endpoint ids, routes, leases, and membership. Join, leave, and migrate register the same id at a new address while every other endpoint keeps running. The step protocol carries correlation ids, retries without executing a step twice, and drops a late reply from a peer that has already been replaced. When a peer dies, every affected step fails over together, and the engine recomputes state that died with the process. Tensor bytes go through a separate engine. connex-rs is the default on RDMA. Mooncake Transfer Engine, UCX, and ZeroMQ remain available.

Two pipeline replicas. Stage 2 of replica 1 dies, in-flight steps move to replica 0, and the stage later rejoins under the same id.
A stage dies with steps in flight. Peers hand those steps to the same stage of another replica, recompute lost KV cache, and, with connex-rs, let the stage rejoin under the same identity.

The picture above is the path we measured. Llama-3-8B, four pipeline stages, A40 GPUs, InfiniBand, two replicas. Killing a stage in the middle of a generation completes every affected request, and a teacher-forced check finds 0 tokens off the argmax. A larger campaign, 72 kill runs and 518,400 tokens, finds the same. With connex-rs, the time from the kill to the next token is 0.01 to 0.83 seconds when four steps are in flight. The same layout on Mooncake Transfer Engine takes 2 to 3 seconds. If a replica is killed every 60 seconds, goodput stays between 99.2 and 99.6 percent. An optional resume proxy, which covers the death of the front-end stage, brings that to 100 percent.

Handover latency and steady-state latency for connex-rs and Mooncake Transfer Engine on the same Llama-3-8B pipeline.
Same cluster, two data planes. Handover drops from seconds to well under a second. Steady-state latency stays together.

Multi-replica pipeline parallelism on vLLM is the path that is integrated and measured. The release is Apache-2.0. Stars, issues, and pull requests are welcome at HeterInfra/connex.

connex-rs asks how the tensor bytes move once a peer has been chosen, and how soon the remaining stages learn that a process has died.

References

2026

  1. SIGCOMM26
    Connex: Endpoint Mobility Primitives for Dynamic LLM Serving
    Yanying Lin, Vincent Liu, Tao Luo, Chengzhong Xu, and Kejiang Ye
    In Proceedings of the ACM SIGCOMM Conference, 2026
    Bib HTML PDF Code Slides Website