ML Systems
Emotion-Conditioned Text-to-Speech
An emotion-conditioned text-to-speech research project exploring controllable generation, speaker preservation, intelligibility, and inference efficiency.
- Python
- PyTorch
- Flow matching
Problem
Controllable emotion conditioning tends to trade off against speaker identity preservation and intelligibility, and small models need to hit reasonable inference speed to be practical.
Constraints
- Evaluation must use speaker-disjoint splits so results reflect generalization, not memorization
- Emotion conditioning shouldn't degrade intelligibility or speaker consistency below usable thresholds
- Sampling cost (steps, precision) needs to be tunable against quality
Approach
Uses a flow-matching generative approach with explicit emotion conditioning, evaluated with speaker-disjoint splits so quality numbers reflect generalization rather than memorized speakers.
System design
Emotion-conditioned TTS pipeline
Text and an emotion/style conditioning signal are encoded jointly, generated via flow matching, and evaluated on intelligibility, speaker consistency, and inference cost.
- Text input + emotion conditioning signal
- Joint text/style encoding
- Flow-matching generation (tunable sampling steps)
- Vocoded audio output
- Evaluation (WER, speaker consistency, real-time factor)
Tradeoffs and limitations
Fewer sampling steps and lower precision improve real-time factor but risk intelligibility and speaker-consistency regressions — the ablations are meant to characterize that curve rather than pick one setting.
Results
This project is still in progress. Planned evaluation includes word error rate, speaker/emotion consistency, and real-time factor across sampling-step and precision ablations. No results are published yet — see the TODO in the project data file.
Next steps
Complete the sampling-step and precision ablations, then publish word error rate, real-time factor, and consistency results once measured.