Juan Pablo Zuluaga

Senior Research Scientist · Agigo AG · Zürich

🇨🇴 🇨🇭

I build and deploy speech-and-audio LLMs, production ASR, and spoken language understanding systems. PhD from EPFL & IDIAP. Previously at Apple and AWS.

scholar github huggingface linkedin email cv

40+ publications Interspeech · ICASSP · EMNLP · SLT · JMLR · TSD

about

I'm a Senior Research Scientist at Agigo AG, a Swiss AI company building autonomous AI agents. My work sits at the intersection of natural language processing and automatic speech recognition, with a strong focus on speech-and-audio LLMs.

I completed my PhD at EPFL and IDIAP in 2024. My thesis tackled automatic speech recognition for air traffic control, one of the hardest real-world ASR domains. Along the way I built the ATCO2 corpus, fine-tuned self-supervised models for this domain, and published work on speaker diarization, speaker role detection, and contextual ASR.

Before Agigo, I interned at Apple (ML for ASR on tail named entities) and at AWS (speech translation and transcription). I hold master's and bachelor's degrees in Mechatronics Engineering from Universidad de Oviedo and Universidad Autónoma del Caribe.

I live in Zürich. Originally from Baranoa, Colombia.

currently

High-throughput LLM & speech model serving: production deployment with vLLM-Omni: streaming decoders, CUDA Graphs, torch.compile, and multi-client inference scheduling.

TTS controllability & steering: conditioning generative TTS on prosody, speaker identity, and style, including voice cloning and zero-shot cross-lingual synthesis.

Full-stack model development: end-to-end: data curation, large-scale synthetic data generation, training optimization targeting 50% MFU, and deployment tuning for low-latency serving.

Local LLM deployment & KV-cache offloading: memory-efficient inference using disaggregated caching and offloading techniques (e.g., LMCache) for constrained-GPU and edge scenarios.

Holistic TTS evaluation pipelines: state-of-the-art automated evaluation covering intelligibility, speaker similarity, naturalness, prosody, and robustness across languages and domains.

Omni-modal data pipelines: large-scale processing with omni LLMs to generate multi-task RL training signal across speech, text, and audio.

engineering

I work across the full stack behind production speech and language AI, from low-level GPU kernels up to high-concurrency serving. Deep systems knowledge paired with modern tooling lets me compress the loop from idea to shipped system, and get outsized leverage from a small team.

CUDA & Triton kernel development TTS systems & controllability LLM & omni-modal inference High-concurrency deployment Large-scale data curation Rapid prototyping & evaluation

selected projects

arXiv2026

GRAFT: Grafted Reference Audio for Fine-Grained Pronunciation in Zero-Shot TTS

Among the first zero-shot TTS systems to give per-word pronunciation control: graft a short spoken hint of a tricky word, a rare name, loanword, or technical term, and GRAFT speaks the whole sentence in a cloned voice with that exact pronunciation.

22–39% lower target-word phoneme error rate

Open source2026

vLLM-Omni: Production Serving for Omni-Modal Models

20+ merged PRs to vLLM-Omni, the production inference engine for text, speech, audio, and vision, focused on streaming TTS, CUDA-graph kernels, and high-concurrency serving.

20+ merged PRs

Agigo AG2025

Production TTS & Speech-LLM Serving at Scale

Building and deploying speech-and-audio LLMs and TTS at production scale: GPU-efficient multi-client inference, synthetic conversational data, and low-latency streaming.

50% MFU training-efficiency target

arXiv2023

ATCO2: The Largest Open Air Traffic Control Speech Corpus

A 5,000-hour open corpus of real air traffic control communications, with transcripts, speaker-role labels, and contextual metadata. The largest open ATC dataset for speech recognition and understanding.

5,000 h of live ATC audio

→ all projects

experience

Senior Research Scientist

Production speech-and-audio LLMs, synthetic conversational data, GPU-efficient multi-client inference for real-world ASR and TTS deployments.

Research Engineer

Speech recognition, understanding, and generation for conversational AI agents.

ML Engineer Intern

Discriminative training of language models for ASR on tail named-entity data.

Applied Scientist Intern

Joint speech-to-text translation and transcription research. Work published at EMNLP 2023.

PhD Researcher

Thesis: Low-Resource Speech Recognition and Understanding for Challenging Applications. Advised by Dr. Petr Motlicek and Prof. Hervé Bourlard.

Research Engineer

ATCO2 project (EU Horizon 2020). Automatic speech recognition and contextual understanding for air traffic control communications.

MSc · Erasmus Mundus EU4M

Mechatronics & Micro-Mechatronics. Fully funded Erasmus Mundus scholarship (EU Commission). Thesis on computer vision for breast cancer diagnosis.

BSc · Mechatronics Engineering

Mechatronics Engineering. DAAD Research Scholarship (Germany, 2014).

→ full cv with education & awards

featured publications

Unifying Global and Near-Context Biasing in a Single Trie Pass

TSD 2025

I. Thorbecke, E. Villatoro-Tello, J. P. Zuluaga, S. Kumar, S. Burdisso, P. Rangappa, A. Carofilis, S. Madikeri, P. Motlicek, K. Pandia, K. Hacioglu, A. Stolcke.

Single-pass trie unifies global vocabulary biasing with utterance-level context biasing for transducer ASR.

paper

@inproceedings{unifyingglobal2025,
  title = {Unifying Global and Near-Context Biasing in a Single Trie Pass},
  author = {I. Thorbecke and E. Villatoro-Tello and J. P. Zuluaga and S. Kumar and S. Burdisso and P. Rangappa and A. Carofilis and S. Madikeri and P. Motlicek and K. Pandia and K. Hacioglu and A. Stolcke},
  booktitle = {TSD},
  year = {2025},
  url = {https://doi.org/10.1007/978-3-032-02548-7_15}
}

Speech Data Selection for Efficient ASR Fine-Tuning

ICASSP 2025

P. Rangappa, S. Madikeri, J. P. Zuluaga, J. Villatoro-Tello, P. Motlicek.

A domain classifier plus pseudo-label filtering cuts ASR fine-tuning compute by ~40% at matched WER.

paper pdf

@inproceedings{speechdata2025,
  title = {Speech Data Selection for Efficient ASR Fine-Tuning},
  author = {P. Rangappa and S. Madikeri and J. P. Zuluaga and J. Villatoro-Tello and P. Motlicek},
  booktitle = {ICASSP},
  year = {2025},
  url = {https://ieeexplore.ieee.org/abstract/document/10888138/}
}

XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models

ICASSP 2025

S. Madikeri, J. P. Zuluaga, P. Rangappa, J. Villatoro-Tello, P. Motlicek.

Streaming ASR atop a frozen self-supervised backbone, without sacrificing non-streaming accuracy.

arxiv

@inproceedings{xlsrtransducer2025,
  title = {XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models},
  author = {S. Madikeri and J. P. Zuluaga and P. Rangappa and J. Villatoro-Tello and P. Motlicek},
  booktitle = {ICASSP},
  year = {2025},
  url = {https://arxiv.org/abs/2407.04439}
}

Open-source Conversational AI with SpeechBrain 1.0

JMLR 2024

M. Ravanelli, T. Parcollet, A. Moumen, S. de Langen, C. Subakan, P. Plantinga, Y. Liao, S. Cornell, D. Roman, S. Moradi, D. Chander, D. Petermann, Y. Wang, J. P. Zuluaga, et al.

Co-authored the 1.0 release of SpeechBrain, a PyTorch toolkit for conversational AI.

paper code

@article{opensource2024,
  title = {Open-source Conversational AI with SpeechBrain 1.0},
  author = {M. Ravanelli and T. Parcollet and A. Moumen and S. de Langen and C. Subakan and P. Plantinga and Y. Liao and S. Cornell and D. Roman and S. Moradi and D. Chander and D. Petermann and Y. Wang and J. P. Zuluaga and others},
  journal = {JMLR},
  year = {2024},
  url = {https://jmlr.org/papers/v25/24-0991.html}
}

End-to-end single-channel speaker-turn aware conversational speech translation

EMNLP 2023

J. P. Zuluaga, Z. Huang, X. Niu, R. Paturi, S. Srinivasan, P. Mathur, B. Thompson, M. Federico.

First end-to-end speech translation system that handles speaker turns and overlapped speech on a single channel.

paper arxiv code

@inproceedings{endto2023,
  title = {End-to-end single-channel speaker-turn aware conversational speech translation},
  author = {J. P. Zuluaga and Z. Huang and X. Niu and R. Paturi and S. Srinivasan and P. Mathur and B. Thompson and M. Federico},
  booktitle = {EMNLP},
  year = {2023},
  url = {https://aclanthology.org/2023.emnlp-main.493/}
}

HyperConformer: Multi-Head HyperMixer for Efficient Speech Recognition

Interspeech 2023

F. Mai, J. P. Zuluaga, T. Parcollet, P. Motlicek.

Replaces Conformer attention with HyperMixer, matching accuracy at a fraction of the compute.

arxiv

@inproceedings{hyperconformermulti2023,
  title = {HyperConformer: Multi-Head HyperMixer for Efficient Speech Recognition},
  author = {F. Mai and J. P. Zuluaga and T. Parcollet and P. Motlicek},
  booktitle = {Interspeech},
  year = {2023},
  url = {https://arxiv.org/abs/2305.18281}
}

CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification

Interspeech 2023 ★ Best Student Paper nominee

J. P. Zuluaga, S. Sarfjoo, A. Prasad, I. Nigmatulina, P. Motlicek, K. Ondrej, O. Ohneiser, H. Helmke.

Accent classification benchmark on Common Voice using large self-supervised models.

arxiv code model

@inproceedings{commonaccentexploring2023,
  title = {CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification},
  author = {J. P. Zuluaga and S. Sarfjoo and A. Prasad and I. Nigmatulina and P. Motlicek and K. Ondrej and O. Ohneiser and H. Helmke},
  booktitle = {Interspeech},
  year = {2023},
  url = {https://arxiv.org/abs/2305.18283}
}

How Does Pre-Trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications

IEEE SLT 2022

J. P. Zuluaga, A. Prasad, I. Nigmatulina, S. Sarfjoo, P. Motlicek, M. Kleinert, H. Helmke, O. Ohneiser, Q. Zhan.

Systematic study of self-supervised pretraining under domain shift, 20–40% relative WER cut on Air Traffic Control.

arxiv code model

@inproceedings{howdoes2022,
  title = {How Does Pre-Trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications},
  author = {J. P. Zuluaga and A. Prasad and I. Nigmatulina and S. Sarfjoo and P. Motlicek and M. Kleinert and H. Helmke and O. Ohneiser and Q. Zhan},
  booktitle = {IEEE SLT},
  year = {2022},
  url = {https://arxiv.org/abs/2203.16822}
}

→ all publications