Testing whether a text-to-speech model already contains prosody controls nobody exposed. Treating inference-time control as an optimisation problem over latent dimensions, with an emotion encoder supplying the objective.
Role Project lead. Period 2025. Code github.com/eryawww/kokoro_hack
Modern text-to-speech models produce speech with intonation, timing and stress that sound deliberate, but they generally expose no control over any of it. You get whatever delivery the model decides on.
The conjecture behind this project is that the control already exists inside the network as a byproduct of training, and simply was never given an interface. If that is true, you should be able to find it and use it without retraining anything.
It largely holds. By treating a small number of latent dimensions as the only free parameters and optimising them against an emotion objective, the delivery of the generated speech can be pushed toward a target emotion while every weight in the speech model stays frozen.
A model that learned to read text aloud from human recordings must have learned to represent how a line is delivered, because the training data varies in delivery and the model reproduces that variation convincingly. Somewhere in the network there is a representation of delivery.
The claim is that this representation is both localised enough to find and continuous enough to move along. That is an empirical question, and the value of the project is in the method for answering it rather than in the conclusion.
I broke it into three steps, and I would use the same three on any similar question. First, identify the candidate latent space where the property might live. Second, determine what transformation of that space corresponds to changing the property. Third, and most easily skipped, define in advance how you will know whether the control worked.
That third step deserves the emphasis. Steering research is unusually easy to fool yourself in, because a human listening to their own output will hear the change they were hoping for.
The measurement problem and the control problem have the same solution here.
Take a Wav2Vec2 model finetuned for emotion recognition. It maps a waveform into a space where emotions are separable. From EmoVDB, an emotional speech dataset, compute a centroid for each target emotion in that space. Now “does this audio sound happy” has a numeric answer: the distance from its embedding to the happy centroid.
That gives an objective, and because the whole path is differentiable, it also gives a method. Fix every weight in the speech model. Treat specific latent dimensions as parameters. Generate audio, encode it, measure cosine embedding loss against the target centroid, and backpropagate all the way to the latent dimensions. Optimise those, and only those.
The pleasing property of this setup is that the evaluator and the optimiser are the same object, which makes the result honest in one narrow sense and suspicious in another. It is honest because there is no human in the loop deciding whether it worked. It is suspicious because a sufficiently aggressive optimiser can find audio that satisfies the emotion encoder without sounding like anything a person would call happy, which is why listening to the output remains part of the process rather than an optional extra.
Almost none of the difficulty was in the idea. It was in making the pipeline differentiable at all.
A shipped text-to-speech model is written to be called, not to be optimised through. The default entry point does tokenisation, phoneme conversion, normalisation and audio assembly in ordinary Python, and every one of those steps breaks the chain that automatic differentiation needs. The code works perfectly and produces no gradient.
So the real work was going into the model’s own code, pulling out the convenience layer, and rebuilding a continuous path from the latent dimensions through to the waveform. That is unglamorous, it produces nothing you can show anyone, and it is the entire reason the rest of the project was possible.
I have come to treat this as a general signal. When a research idea seems easy and stalls anyway, the obstruction is frequently an interface written for a different purpose rather than anything conceptual.
Whether the dimensions found this way are the same across voices, which would say something about whether the representation is a property of the model or of the particular speaker embedding.
And whether optimised latents transfer as fixed offsets, so that “happier” becomes a vector you add rather than an optimisation you rerun. That is the difference between a research result and a usable control.
Complete as an experiment, public as code. Not packaged as a library, and the results are qualitative rather than benchmarked against a published baseline.