Conditioning and Video: Giving Voice to the Ink
We know how the water mage pulls ink out of dark water to reveal a painting. But what if we don’t want just any random painting? What if we want a specific warrior riding across a bridge under a stormy sky? And what if we want to watch him actually gallop?
A master storyteller steps up beside the bowl. As the mage dips his brush to pull out the ink, the storyteller whispers into the water: “A solitary samurai… galloping through thunder.”
The whispered words ripple across the surface. The brush listens. It steers its hundreds of tiny, blind guesses toward the sound of the voice, aligning every extracted drop of ink with the words spoken.
Then, the storyteller begins to chant continuously in a steady rhythm. Instead of clearing just a single bowl, the mage pulls ink across a continuous reel of unrolling parchment scrolls. Because each frame shares the memory of the frame before it, the ink doesn’t freeze into a static portrait—it surges, breathes, and flows into motion. The samurai rides.
The Reality
By default, a diffusion model generates random samples from everything it has ever learned. To control the output, we use Conditioning.
When you type a text prompt, an encoder (like CLIP or T5) converts your words into mathematical concept vectors. Through a mechanism called Cross-Attention, the diffusion network checks these text vectors at every single denoising step, nudging the pixels so the emerging image matches your description.
To create Video (like Sora or Gen-3), AI extends spatial diffusion across time (Spatio-Temporal Diffusion). Instead of generating isolated 2D images, the model denoises a 3D block of frames simultaneously. Temporal attention layers ensure that objects, lighting, and physics remain consistent from one millisecond to the next, turning static noise into fluid, cinematic video.
The Why
Conditioning transforms generative AI from an unpredictable lottery into a precise creative tool. Temporal video diffusion takes it a step further: it proves that models aren’t just learning how objects look—they are beginning to model how the physical world behaves over time.
The Takeaway
Random creation is a novelty; controlled, coherent evolution across time is true art.
AI specialists call it: Cross-Attention Conditioning / Classifier-Free Guidance (CFG) / Spatio-Temporal Diffusion
Text conditioning guides the reverse process by injecting prompt embeddings c into the network via cross-attention layers. Classifier-Free Guidance blends unconditional and conditional score estimates ε_θ(z_t, t, c) to amplify prompt adherence: ε~ = ε_θ(z_t, t, ∅) + s · (ε_θ(z_t, t, c) - ε_θ(z_t, t, ∅)). In video diffusion, 3D convolutions or temporal attention modules process sequence latents z ∈ R^(B×T×C×H×W), enforcing temporal consistency across frames.
💬 If you could generate a 5-second video of any moment in your memory or history to see in motion, what would you ask for?
Part 14 of 14 | #GenerativeModelsForHumans
#ai_edu Based on Stanford and industry lectures