AI cloned me: how I made an entire short with my cloned voice

In short: at the end of «La IA me clonó» ("AI cloned me") there's a short about Terraform that I didn't record myself: the voice is a clone of mine, an AI programmed the animation and the subtitles synced on their own. Here's how each part was made and what I decided along the way.

A short made with AI, end to end

The short you see at the end of the video explains what Terraform is with an Azure virtual machine that gets created, modified and destroyed on screen. There's no camera and no microphone: it came out of four pieces made with AI.

1. A script written line by line.
2. My cloned voice, reading that script.
3. An animation built with code.
4. A sync step that makes the animation and the subtitles follow the voice.

My part was deciding what gets said, picking the takes and approving every version.

The script: every line with its intent

If you hand a voice engine a whole paragraph, it reads all of it in the same tone, and that's exactly what makes an AI voice sound flat.

That's why the script goes line by line, and every line carries its intent: it asks, explains, warns or gets excited. A few lines from the Terraform short (it's in Spanish, translated here):

The intent decides the tone each line is generated with: the warning goes slower, the question goes up.

Every line also carries its keywords, the ones that trigger the animation: when the voice says "plan", the command shows up.

The voice: my clone, line by line

Cloning without training anything

You don't need to train a model on hours of recordings to clone your voice. It's cloned zero-shot: the model already knows how to speak, and you give it a reference clip a few seconds long. It copies your timbre, your rhythm, your accent and even how your microphone sounds, and applies all of it to any new text. The catch: the clone is only as good as the clip you give it.

How to record the clips

What changed the result the most:

• Record with the microphone you use for your videos. The model copies the microphone, not just your voice. I record my videos with AirPods, so I recorded the reference with the same AirPods.
• Turn off noise suppression. It clips your breaths and the ends of your words, and the clone learns to speak that way.
• One clip per intent. The clone copies your tone too. If you record a single calm clip, even the warnings come out calm. I recorded four clips of 15 to 25 seconds, one per intent, and every line is generated with its own.
• Speak with your own accent. When reading out loud, people tend to "neutralize" their accent. If you do, the clone learns an accent that isn't yours.

The engine

I use CosyVoice 3, an open source model from FunAudioLLM that speaks Spanish, among other languages. It's licensed under Apache-2.0, which allows commercial use. I run it with VoiceStudio, an open source app that puts an interface on top of several voice engines and runs locally: your voice never leaves your machine. An NVIDIA GPU with 4 or 5 GB of free memory is enough.

Watch out for the default engine. Every VoiceStudio engine has its own license, and the default one is non-commercial. For a monetized channel, pick CosyVoice 3 and check the license of any other engine.

An alternative with an MIT license and Spanish support is Chatterbox Multilingual, by Resemble AI.

The tricks that make it sound like me

• Several takes per line. Every generation comes out a little different. I generate three or more per line, listen to them and keep the best one. That's the difference between "sounds like AI" and "sounds like me".
• A pronunciation dictionary. The engine read "Terraform" with the stress in the wrong place. Before generating, every technical term is replaced with how I actually say it: in Spanish, Terraform becomes «Térraform» and AWS becomes «a doble be ese». In the subtitles the word stays spelled correctly.
• Numbers spelled out. CosyVoice doesn't handle numbers well in Spanish: depending on how you use it, it reads them however it wants or switches them to English, and «3 réplicas» ends up as «three réplicas». I write «tres réplicas».
• Careful with yes-or-no questions. In Spanish, a question like «¿Qué pasa con tus datos?» ("What happens to your data?") goes down at the end, but «¿Terraform se da cuenta?» ("Does Terraform notice?") goes up. Those are the ones that fail the most: if the take comes out flat, I generate another one or rewrite the line.

The animation: code written by AI

The animation wasn't edited: it's built with code, using Remotion, a library that makes videos with React: every scene is a component that renders according to the current moment in the video. It's free for personal use and for companies of up to three people, commercial use included.

I didn't write that code by hand. Claude Code, an AI coding agent, wrote it using the script as the spec: every scene in the script is a scene in the video, and every keyword is a change on screen. "Plan" brings in the command, "destroys" paints the machine red, "gets lost" turns it off.

My job there was a director's: watch every version and ask for changes. And since it's code, if a line changes nothing gets re-edited by hand: the video is simply rendered again.

The sync: the voice leads

With the voice ready, what's left is knowing when each word is said. For that I use Whisper, OpenAI's open source transcription model: it listens to the audio and returns every word with its start and end time. It's the same model I use, through whisper.cpp, so that my AI assistant on Meta Quest 3 understands what I say.

With those timestamps, the animation kicks in right when the voice says each keyword, and the subtitles move at the same pace. Since Whisper sometimes mishears things, the subtitle text comes from the script: only the timing comes from Whisper. And if I regenerate a line, the animation and the subtitles rearrange themselves.

What I decided

AI did the heavy lifting; the decisions were mine:

• What gets said. I approved the script before a single line was generated. I asked for it to open straight with the topic, and we rewrote a line that sounded off to me.
• Which take stays. I listened to the candidates for the doubtful lines and picked one. When none of them worked, we generated more.
• How each term is pronounced. Out of several versions, I picked the one that sounded the way I say it.
• When it's done. I watched every version of the video in full before approving it.

AI produces; the judgment of what's good is yours.

Disclosing that it's AI

• Use each platform's label. TikTok and Instagram require you to turn on their AI-generated content label when the audio realistically imitates a real person's voice, even if it's your own. YouTube doesn't require it if you clone your own voice for a voiceover, but it has the "AI usage" option in YouTube Studio, and I check it anyway.
• Only clone your own voice, or the voice of someone who gave you permission. Without consent, it may be illegal depending on where you live.
• Keep the watermark. VoiceStudio adds an inaudible watermark to everything it generates, turned on by default, which makes it possible to detect that the audio is synthetic.

Wrapping up

If the Terraform example made you want to try it with Azure without paying anything, I show you how with this local Azure emulator. If you're ready for something more real, here's a full architecture on Azure Container Apps with Terraform.

I keep making shorts like this one, about DevOps and cloud. I post them on YouTube, TikTok and Instagram. And if you give voice cloning a try, tell me how it went.