While running LLM inference on a cluster, we kept hitting limits in NVIDIA NCCL under dynamic serving. A communicator is expensive to manage, runtime communication errors are common, and adding or removing a GPU usually means restarting the whole inference system. The SIGCOMM’26 system is now open source: Connex (HeterInfra/connex).

Connex gives each endpoint a stable logical identity. The process behind that identity can join, leave, fail, or move to a new address while steps are still in flight. Multi-replica pipeline parallelism on vLLM is integrated and measured. Prefill-decode disaggregation has the primitives in the library. An engine integration for that path is still ahead.

Tensor bytes move through a separate data plane. On hosts with RDMA, the default is connex-rs (HeterInfra/connex-rs), a Rust engine over UCX. On the same Llama-3-8B pipeline, steady-state latency matches Mooncake Transfer Engine. After a stage is killed, the time to the next token drops from 2 to 3 seconds with Mooncake to well under a second. Stars, issues, and pull requests are welcome.