Respan Releases Open-Source Rust AI Gateway After Abandoning LiteLLM Wrapper

Why Respan Rebuilt Its Gateway

Respan released an open-source AI gateway written in Rust on Tuesday after two years and more than 100,000 lines of wrapper code around LiteLLM. The breaking point came in January 2026, when a routine LiteLLM upgrade caused intermittent connection failures. Respan rolled back but never found the cause, and engineers since have stopped upgrading LiteLLM directly, cherry-picking upstream fixes one at a time.

A Small Core With Plugins

The gateway borrows from Nginx, keeping the core small and moving routing, model providers, and security into plugins. Developers can build custom plugins using the same interfaces without touching core code. Respan says its Mistral provider takes just 22 lines, and quirks like Groq reporting token usage in a different field are handled separately.

Requests over HTTP or WebSocket pass through plugins in registration order. Each plugin can select a model, modify the response, make additional calls, or stop the request before it reaches a provider. When both sides use the same format, provider-specific fields are preserved. The gateway also translates between protocols, letting an application built for OpenAI's Realtime API connect to Gemini Live without client changes, though some features may not carry over.

Compile-Time Plugins Raise Questions

Custom plugins are compiled into the gateway, so adding one means maintaining code and rebuilding the binary. Because plugins run inside the gateway process, they can access model requests and other sensitive data. Rust prevents memory-safety bugs but cannot guard against malicious dependencies or flawed logic, a concern raised in Hacker News discussion. One commenter questioned whether compiled crates should be called plugins, citing build pipeline overhead and supply-chain risks. Respan's founder acknowledged the concern, noting developers can inspect the source they compile.

Plugin order also matters. Plugins run in registration order, and one can answer or reject a request before others execute. A cache placed before a security check could let a cached response bypass that check entirely.

LiteLLM Also Moving to Rust

LiteLLM is porting its gateway to Rust one route at a time, with the Rust core off by default and requests falling back to Python when needed. The company outlined the plan in June and aims for a pure Rust server handling all hot paths, auth, and rate limiting. LiteLLM has published benchmarks claiming its Rust gateway has about seven times less overhead and nine times less memory than Bifrost, though those figures come from LiteLLM's own tests.

Respan's gateway is still pre-alpha. Developers can download a prebuilt binary from GitHub without the paid platform, though custom plugins require compiling their own build.

Sources: The New Stack