The systems, datasets, models, and open-source software I’m most proud of, at a glance. Each links to its paper or code.
Projects
arXiv2026
GRAFT: Grafted Reference Audio for Fine-Grained Pronunciation in Zero-Shot TTS
Among the first zero-shot TTS systems to give per-word pronunciation control: graft a short spoken hint of a tricky word, a rare name, loanword, or technical term, and GRAFT speaks the whole sentence in a cloned voice with that exact pronunciation.
22–39% lower target-word phoneme error rate#1 human eval on pronunciation accuracy6 languages, incl. EN, FR, DE, ES, IT
Spoken hintshort audio of the word
Graft at word positionspeech-token anchoring
Voice conversionhint voice ≠ target voice
Cloned-voice speechcorrect pronunciation
Fixes a core failure of zero-shot TTS, mispronouncing rare names, loanwords, and technical terms, by letting anyone specify pronunciation with a quick audio example instead of phonemes.
@misc{graftgrafted2026,
title = {GRAFT: Grafted Reference Audio for Fine-Grained Pronunciation in Zero-Shot TTS},
author = {A. Asonitis and F. Verdini and A. Farhadipour and V. Avijeet and P.-E. Honnet and M. Razavi and J. P. Zuluaga},
year = {2026},
eprint = {2607.02633},
archivePrefix = {arXiv}
}Open source2026
vLLM-Omni: Production Serving for Omni-Modal Models
20+ merged PRs to vLLM-Omni, the production inference engine for text, speech, audio, and vision, focused on streaming TTS, CUDA-graph kernels, and high-concurrency serving.
20+ merged PRsQwen3-TTS & OmniVoice pathsprod high-concurrency serving
Text request
Code Predictorautoregressive
Code2Wav decoderCUDA Graph · Triton
Streaming audiolow TTFA
Makes open, omni-modal TTS serving production-ready: faster, streamable, and stable at concurrency.
Agigo AG2025
Production TTS & Speech-LLM Serving at Scale
Building and deploying speech-and-audio LLMs and TTS at production scale: GPU-efficient multi-client inference, synthetic conversational data, and low-latency streaming.
50% MFU training-efficiency targetmulti-client concurrent production servingstreaming low-latency TTS
Speech-LLM + TTS models
GPU-efficient batchingCUDA graphs · torch.compile
Multi-client schedulerhigh concurrency
Streaming audiolow latency
Turns research-grade speech models into reliable, low-latency products serving many clients concurrently, the core of my work at Agigo.
arXiv2023
ATCO2: The Largest Open Air Traffic Control Speech Corpus
A 5,000-hour open corpus of real air traffic control communications, with transcripts, speaker-role labels, and contextual metadata. The largest open ATC dataset for speech recognition and understanding.
5,000 h of live ATC audio4-year EU project (SESAR JU · Horizon 2020)1st open ATC corpus at this scale
Live VHF audioEuropean airspace
Auto-transcribe & filterpseudo-labeling
Role labels + metadatacallsigns, commands
5,000 h corpusopen release
Turned a data-starved field into one where large pretrained models can be studied at realistic scale. Now a standard benchmark, available through ELRA.
@misc{atco2the2023,
title = {ATCO2: The Largest Open Air Traffic Control Speech Corpus},
author = {J. P. Zuluaga and K. Vesely and I. Szoke and P. Motlicek and M. Kocour and M. Rigault and K. Choukri and A. Prasad and S. Sarfjoo and I. Nigmatulina and J. Cernocky and D. Klakow},
year = {2023},
eprint = {2211.04054},
archivePrefix = {arXiv}
}JMLR2024
SpeechBrain 1.0: Open-Source Conversational AI Toolkit
Co-authored the 1.0 release of SpeechBrain, the PyTorch toolkit for speech and conversational AI, used by dozens of research groups and startups.
1.0 stable release (JMLR 2024)ASR·TTS·SV unified reproducible recipesOSS used across academia & industry
ATC ASR recipeswav2vec2 · XLS-R
HyperConformerefficient encoder
CommonAccentaccent ID benchmark
into SpeechBrain 1.0
Unifies ASR, TTS, speaker recognition, and dialogue into one PyTorch toolkit with reproducible recipes, lowering the barrier to speech research.
@article{speechbrain102024,
title = {SpeechBrain 1.0: Open-Source Conversational AI Toolkit},
author = {M. Ravanelli and T. Parcollet and A. Moumen and S. de Langen and C. Subakan and P. Plantinga and Y. Liao and S. Cornell and J. P. Zuluaga and others},
journal = {JMLR},
year = {2024},
url = {https://jmlr.org/papers/v25/24-0991.html}
}Interspeech2023★ Best Student Paper nominee
CommonAccent: Accent Classification with Self-Supervised Models
A benchmark and open model for spoken-accent classification on Common Voice, reaching 95%+ accuracy with self-supervised models.
95.1% accent-ID accuracy (English)★ Best Student Paper nominee4 languages benchmarked
Speech utteranceCommon Voice
Self-supervised encoderwav2vec 2.0-XLSR
Stat-pooling headaccent classifier
Accent label95%+ accuracy
Showed that self-supervised speech representations transfer strongly to paralinguistic tasks, not just transcription.
@inproceedings{commonaccentaccent2023,
title = {CommonAccent: Accent Classification with Self-Supervised Models},
author = {J. P. Zuluaga and S. Sarfjoo and A. Prasad and I. Nigmatulina and P. Motlicek and K. Ondrej and O. Ohneiser and H. Helmke},
booktitle = {Interspeech},
year = {2023},
url = {https://arxiv.org/abs/2305.18283}
}IEEE SLT2022
Self-Supervised ASR for Air Traffic Control
Fine-tuning self-supervised speech models (wav2vec 2.0 / XLS-R) for one of the hardest real-world ASR domains, air traffic control, cutting word error rate by 20–40% over supervised baselines.
20–40% relative WER reduction~6% WER on ATCOSIM3 open models on HuggingFace
Unlabeled speechthousands of hours
wav2vec 2.0 / XLS-Rself-supervised
Fine-tune on ATCATCOSIM · LDC · UWB
Robust ATC ASR20–40% WER ↓
Among the first studies to show how self-supervised pretraining transfers to a safety-critical, heavily domain-shifted setting. The released models are a practical baseline for ATC ASR.
@inproceedings{selfsupervised2022,
title = {Self-Supervised ASR for Air Traffic Control},
author = {J. P. Zuluaga and A. Prasad and I. Nigmatulina and S. Sarfjoo and P. Motlicek and M. Kleinert and H. Helmke and O. Ohneiser and Q. Zhan},
booktitle = {IEEE SLT},
year = {2022},
url = {https://arxiv.org/abs/2203.16822}
}EMNLP2023
End-to-End Speaker-Turn-Aware Speech Translation
The first end-to-end speech translation system that handles speaker turns and overlapping speech on a single channel. Built during my internship at AWS.
1st single-channel, speaker-turn-aware E2E STAWS internship, EMNLP 2023open code released
Single-channel audiooverlapping speakers
End-to-end modeljoint turns + translation
Per-speaker translationwho said what
Pushed speech translation toward realistic conversational audio, and showed that jointly modeling speaker structure and translation beats cascading them.
@inproceedings{endto2023,
title = {End-to-End Speaker-Turn-Aware Speech Translation},
author = {J. P. Zuluaga and Z. Huang and X. Niu and R. Paturi and S. Srinivasan and P. Mathur and B. Thompson and M. Federico},
booktitle = {EMNLP},
year = {2023},
url = {https://aclanthology.org/2023.emnlp-main.493/}
}