Text to Speech with Emotions

How emotion works in AudioDrama

Most text-to-speech tools expose emotion as a dropdown for the whole text: “happy”, “sad”, “neutral”. AudioDrama works per line, because that is how actors are directed. You write the instruction next to the words it applies to.

There are two places a direction can go:

• After the name: Mara [whispering]: Did you see that? The direction applies to the whole line.
• Inside the line: Mara: Did you see that? [short pause] Someone is up there. Use this to change delivery partway through, or to add a pause.

Only one bracket group fits between the name and the colon. If you need more than one direction there, combine them in one pair of brackets with a comma ([angry, through gritted teeth]) or move the rest after the colon.

The model is Fish Audio's Drama 3 text-to-speech model, which accepts these natural-language instructions. AudioDrama is not affiliated with Fish Audio.

The same line, three performances

The quickest way to understand directions is to hold the words still and change only the bracket. These three lines are identical apart from the direction:

Mara [whispering]: You said you'd be home by nine.
Mara [angry, through gritted teeth]: You said you'd be home by nine.
Mara [crying quietly]: You said you'd be home by nine.

Listen to the sample: one is a secret, one is a threat, one is grief. Nothing about the text changed. That recording is a single, unedited take; results vary by voice and direction, so treat your first take as a draft.

Directions to start with

These are the directions offered as one-click buttons in the app. They are starting points, not a complete list.

The <emphasis>…</emphasis> tag stresses a single word, as in It's <emphasis>ours</emphasis>.

Writing directions that work

There is no guarantee for any single direction, but these habits give the model the clearest instructions:

Be specific and short

Two or three words beat a sentence. [tired, half asleep] is clearer than [sounding like someone who has not slept in a long time], and it costs fewer bytes.

Add one attribute for pace or volume

Emotion is only half of delivery. Pairing a feeling with a manner, such as [angry, speaking fast] or [sad, slow], narrows the result.

Split lines when the emotion changes

One line with [laughing] and then a sudden [serious] asks a lot. Two lines from the same character, each with its own direction, are easier to get right.

Do not stack contradictions

[whispering, shouting] gives the model nothing to follow. Choose one.

Test short, then run the whole scene

Every generation uses credits. Try the key lines first, find directions you like, then generate the scene.

An escalating scene

Here is a five-line scene that uses direction, a pause and emphasis together. It is 386 bytes with speaker tags, so at least 26 seconds of credit.

Dev [calm, friendly]: Hey. Did you get a chance to look at the budget?
Ravi [defensive, speaking fast]: I told you, it's handled. It's handled.
Dev [slow, deadly serious]: Ravi. Where did the money go?
Ravi [sigh]: [short pause] Okay. I spent it. All of it.
Dev [quiet, hurt]: It's not <emphasis>my</emphasis> money. It's <emphasis>ours</emphasis>.

Note how each line carries a different attitude, and how Ravi's [short pause] sits inside his line after the sigh. Scenes like this are where per-line direction pays off. For the full script rules, see the guide to the audio drama script format.

Honest limits

• Interpretation, not control. Directions are suggestions to a model. The same direction can sound different on different voices.
• No fine tuning. No intensity slider, no SSML, no phoneme editing, no timing control beyond pauses and punctuation.
• English voices. The 16 built-in voices are English. Directions in other languages may work but are not guaranteed.
• Voice only. No sound effects and no music. A direction like [door slams] will not make a door slam.
• Size limits. 6,000 characters per script and a 9,000-byte cap per job, including directions. Free accounts get 2 minutes once; see pricing.

For multi-character casts, read the radio play generator page to see how emotion and scene-setting work together.