Diffusion & Generative-Media Models
Parent: LLM Models and APIs · researched 2026-06-03T00:22:18.789Z· 18 sources · 12 concepts · skill diffusion-generative-media
The model family that generates continuous media (images, video, audio) by learning to reverse a noising process. A model-layer reference under the ai-agent-engineering hub (2024–2026). This is the ge
Diffusion & Generative-Media Models — Image / Video / Audio
- The model family that generates continuous media (images, video, audio) by learning to reverse a noising process. A model-layer reference under the ai-agent-engineering hub (2024–2026). This is the generation side and is deliberately separate from multimodal-llm-architecture (which is media understanding - vision encoders, VLMs) and from da-35-synthetic-data-generation (which is tabular synthesis). When the task is "make a picture/video/sound," it lives here. [source]
1. The denoising-diffusion core (DDPM)
- A forward process gradually adds Gaussian noise to data over T steps until it is ~pure noise; a neural network learns the reverse (denoising) process. Training minimizes a simple objective: predict the noise added at a random timestep. [source]
- Parameterizations: predict the noise ε (DDPM), the data x₀, or the v (velocity) target. v-prediction is preferred at high noise/high resolution and for distillation stability. [source]
- Noise schedule: linear, cosine (Nichol & Dhariwal - better for high-res), or the continuous σ-space of EDM. The schedule controls how SNR decays and matters a lot for quality. [source]
- DDPM is the foundation; everything below is either a faster sampler, a better parameterization/space, a better architecture, or a control method on top. [source]
2. The score-based / SDE view
- Song & Ermon's score-based generative models unified diffusion under stochastic differential equations: the reverse process integrates the score (∇ₓ log p(x)) learned by the network. [source]
- VP-SDE (variance-preserving ≈ DDPM) and VE-SDE (variance-exploding ≈ NCSN). [source]
- The probability-flow ODE: a deterministic ODE with the same marginals as the SDE - enables fast deterministic sampling and exact likelihoods, and is the bridge to flow matching. [source]
- EDM (Karras et al.): a cleaner design space - σ-parameterized noise, preconditioning of network in/out, and the Heun sampler; "EDM2" refines training dynamics. EDM's framing is the modern default mental model. [source]
3. Latent diffusion (why almost everything runs in latent space)
- Pixel-space diffusion is expensive. Latent Diffusion (Rombach et al. → Stable Diffusion) runs the diffusion process in the compressed latent space of a pretrained VAE: encode image → diffuse/denoise the latent → decode. This cut compute ~10–100× and made open text-to-image practical. The VAE's quality (and its KL/VQ regularization) bounds the system's fidelity; the denoiser is conditioned on text via cross-attention to a text encoder (CLIP/T5). [source]
4. Architectures: UNet → Diffusion Transformer
- The original denoiser is a UNet (conv encoder–decoder + skip connections + attention at low resolutions). [source]
- Diffusion Transformer (DiT) replaces the UNet backbone with a transformer over latent patches - better scaling with parameters/compute, variable-length handling, and reuse of LLM-stack techniques. DiT is now dominant for frontier models. [source]
- MMDiT (multimodal DiT, SD3) runs separate but interacting streams for text and image tokens; FLUX is a large rectified-flow DiT. The throughline: frontier image/video models are DiTs trained with flow matching (next section). [source]
5. Flow matching & rectified flow
- A simpler, often-superior alternative training framework that has largely won at the frontier: [source]
- Continuous Normalizing Flows / Flow Matching (Lipman et al.): regress a time-dependent velocity field that transports noise to data along a probability path. Conditional FM makes the objective tractable (regress to a per-sample conditional velocity). [source]
- Rectified Flow (Liu et al.): learn straight transport paths between noise and data - straighter paths integrate in fewer steps. Reflow iteratively straightens. [source]
- Stochastic interpolants generalize diffusion + FM under one theory. [source]
- FM/RF improve sample quality and few-step generation and are the training objective behind SD3 and FLUX. Diff2Flow (CVPR 2025) shows diffusion and FM are close enough to fine-tune a diffusion prior as an FM model by rescaling timesteps - evidence of the paradigms' convergence. [source]
6. Guidance — classifier & classifier-free
- Classifier guidance (Dhariwal & Nichol) steers sampling with a separate classifier's gradient. [source]
- Classifier-free guidance (CFG) (Ho & Salimans) is the workhorse: jointly train conditional + unconditional (drop the prompt p% of the time), then at sampling extrapolate ε = ε_uncond + s·(ε_cond − ε_uncond). The guidance scale s trades prompt-adherence vs diversity/fidelity; negative prompts put content in the unconditional branch to push it away. [source]
- Costs/caveats: CFG doubles per-step compute (two evals); high s over-saturates - hence CFG rescaling (Lin et al.) and guidance-distillation (§8) to fold CFG into one eval. [source]
7. Samplers / schedulers (the steps↔quality knob)
- The sampler numerically integrates the reverse ODE/SDE; fewer steps = faster but lower quality: [source]
- DDIM - deterministic, non-Markovian; enables 20–50-step sampling and latent interpolation/inversion. [source]
- DPM-Solver / DPM-Solver++ - high-order ODE solvers; good quality at ~10–20 steps (a common default). [source]
- Euler / Euler-a / Heun, UniPC, Karras σ schedule for step spacing. [source]
- Rule of thumb: solver choice + step count + schedule jointly set the speed/quality frontier before you reach for distillation. [source]
8. Few-step generation & distillation
- Pushing from ~20–50 steps down to 1–4: [source]
- Consistency Models (Song et al.) - learn a function mapping any point on a trajectory directly to its origin; sample in 1–2 steps. Latent Consistency Models (LCM) bring this to Stable Diffusion; LCM-LoRA is a plug-in accelerator. [source]
- Distillation families: progressive distillation (halve steps repeatedly), guidance distillation (bake CFG into one eval), consistency distillation, adversarial (ADD/SDXL-Turbo, LADD latent ADD) for 1–4-step high-quality, and sCM (score-regularized continuous-time consistency) for large-scale. [source]
- Video: TurboDiffusion (ShengShu/Tsinghua, 2025) reports ~100–200× end-to-end speedups via step distillation (rCM) + low-bit SageAttention/sparse-linear attention + W8A8 - enabling near-real-time video. [source]
9. Conditioning & control (the six levers)
- Customizing/conditioning a frozen or lightly-tuned base model: [source]
- ControlNet - clones the encoder to add spatial conditioning (edges, depth, pose, segmentation) while locking the pretrained backbone. [source]
- T2I-Adapter - lighter-weight spatial conditioning. [source]
- IP-Adapter - image-prompt (reference-image) style/content conditioning via decoupled cross-attention. [source]
- DreamBooth - fine-tune on a few images to bind a subject to a token (with class-preservation loss). [source]
- LoRA - low-rank adapters for cheap style/subject fine-tuning (the dominant community method); composable. [source]
- Textual Inversion - learn a new embedding ("a new word") for a concept without touching weights. [source]
- (Note: LoRA/DreamBooth for diffusion live here; LoRA/PEFT for LLM text is llm-fine-tuning-peft.) [source]
10. Text-to-video & the video-diffusion stack
- Architecture: latent video DiT with spatiotemporal attention (full 3D, or factorized spatial+temporal); a 3D/causal VAE compresses time as well as space. Conditioning and CFG carry over from image diffusion. [source]
- The hard problem is temporal consistency (flicker, identity drift, motion coherence) and cost (sequence length explodes with frames). [source]
- Landscape (2025–26): closed - Sora/Sora 2, Google Veo 2/3, Kling, Runway Gen-3, Pika, Luma, Minimax/Hailuo. Open - CogVideoX, Mochi-1, HunyuanVideo, Wan (Wan2.x), LTX-Video/LTX-2, Allegro. Diffusers documents the open stack. [source]
- Control for video: WanVideo + ControlNet, image-to-video conditioning, motion LoRAs. [source]
11. Audio, music & other modalities (brief)
- Diffusion also drives audio/music generation (e.g. Stable Audio, audio latent diffusion) and 3D/robotics (diffusion policies for manipulation). The same core - denoise in a learned latent, condition via cross-attention, guide with CFG - transfers across modalities. Deep audio/music modeling is out of scope here; this is the connective overview. [source]
12. Evaluation & efficiency
- Image: FID (Fréchet Inception Distance - distribution match), CLIPScore (prompt alignment), FID-CLIP trade-off curves, increasingly human-preference models (PickScore, ImageReward, HPS). [source]
- Video: FVD (Fréchet Video Distance), VBench dimensions (temporal flicker, motion, subject consistency), human eval. [source]
- Efficiency is its own active survey area (TMLR/TPAMI 2025 efficient-diffusion surveys): architecture, sampler, distillation, and quantization axes - know which axis you're optimizing before reaching for the next trick. [source]
Sources
- Efficient Diffusion Models: A Survey - TMLR 2025 (AIoT-MLSys-Lab); Efficient Diffusion Models - TPAMI 2025 (TsinghuaC3I) [source]
- Video Diffusion Models Survey (2025); HuggingFace, State of open video generation models in Diffusers [source]
- Diff2Flow: Training Flow Matching Models via Diffusion Model Alignment - CVPR 2025 (CompVis) [source]
- On Distillation of Guided Diffusion Models - arXiv 2210.03142; Large-Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency (sCM) - arXiv 2510.08431 [source]
- TurboDiffusion: Accelerating Video Diffusion Models by 100–200× - arXiv 2512.16093 (ShengShu / Tsinghua) [source]
- Six Ways to Control Style and Content in Diffusion Models - Towards Data Science; Understanding and Training IP-Adapters - Mercity Research [source]
- Adaptive Video Distillation: Mitigating Oversaturation and Temporal Collapse - arXiv 2603.21864 [source]
- Foundational (pre-cutoff canon): DDPM (Ho 2020), score-SDE (Song 2021), Latent Diffusion/Stable Diffusion (Rombach 2022), CFG (Ho & Salimans 2022), EDM (Karras 2022), Rectified Flow (Liu 2022), Flow Matching (Lipman 2023), Consistency Models (Song 2023), DiT (Peebles & Xie 2023), SD3/MMDiT (Esser 2024) [source]
Children
- Denoising diffusion core (DDPM, ε/v-prediction, noise schedules) (frontier)
- Score-based SDE/ODE view (VP/VE, probability-flow ODE, EDM/Karras) (frontier)
- Latent diffusion (VAE + denoiser, Stable Diffusion) (frontier)
- Architectures: UNet → Diffusion Transformer (DiT, MMDiT/SD3, FLUX) (frontier)
- Flow matching & rectified flow (conditional FM, stochastic interpolants, Diff2Flow) (frontier)
- Classifier-free guidance (CFG, guidance scale, negative prompts) (frontier)
- Samplers/schedulers (DDIM, DPM-Solver++, Euler/Heun, Karras sigmas) (frontier)
- Few-step generation & distillation (consistency models, LCM, ADD/Turbo, sCM, TurboDiffusion) (frontier)
- Conditioning & control (ControlNet, T2I-Adapter, IP-Adapter, DreamBooth, LoRA, textual inversion) (frontier)
- Text-to-video & video diffusion (spatiotemporal DiT, Sora/Veo/Kling/CogVideoX/Wan, temporal consistency) (frontier)
- Audio/music & other-modality diffusion (overview) (frontier)
- Evaluation & efficiency (FID/FVD/CLIPScore, VBench, efficiency surveys) (frontier)
Frontier under this node: Architectures: UNet → Diffusion Transformer (DiT, MMDiT/SD3, FLUX), Audio/music & other-modality diffusion (overview), Classifier-free guidance (CFG, guidance scale, negative prompts), Conditioning & control (ControlNet, T2I-Adapter, IP-Adapter, DreamBooth, LoRA, textual inversion), Denoising diffusion core (DDPM, ε/v-prediction, noise schedules), Evaluation & efficiency (FID/FVD/CLIPScore, VBench, efficiency surveys), Few-step generation & distillation (consistency models, LCM, ADD/Turbo, sCM, TurboDiffusion), Flow matching & rectified flow (conditional FM, stochastic interpolants, Diff2Flow), Latent diffusion (VAE + denoiser, Stable Diffusion), Samplers/schedulers (DDIM, DPM-Solver++, Euler/Heun, Karras sigmas), Score-based SDE/ODE view (VP/VE, probability-flow ODE, EDM/Karras), Text-to-video & video diffusion (spatiotemporal DiT, Sora/Veo/Kling/CogVideoX/Wan, temporal consistency)