vLLM-Omni is the production inference engine for omni-modality models, text, speech, audio, and vision. I’m an active contributor with 20+ merged PRs, primarily on the Qwen3-TTS and OmniVoice serving paths.
What I work on
Streaming & low-latency TTS: real-time token-by-token audio output [1], dynamic TTFA tuning based on Code2Wav load [2], and flexible initial-phase chunking to minimize perceived latency [3].
GPU kernel optimization: CUDA Graph capture of Code2Wav and CodePredictor [4], Triton SnakeBeta kernel fusion [5], torch.compile with reduce-overhead and static shapes [6], and batched waveform decoding for high-concurrency throughput [7].
Voice cloning & speaker management: global speaker cache manager with LRU eviction [8], OmniVoice voice cloning [9], and reference audio upload with optional transcription [10].
Prefix-cache hardening: OOM guards and hot-buffer fallbacks [11], and fixes for silent cache corruption under long-context requests [12].
Why it matters
Serving TTS at production scale is different from pure-LLM serving: each request runs an autoregressive code-predictor and a waveform decoder (Code2Wav), each with its own batching and latency budget. This work makes that pipeline fast, streamable, and stable under real-world concurrent load.