# Diffusion & Generative-Media Models > The model family that generates continuous media (images, video, audio) by learning to reverse a noising process. A model-layer reference under the ai-agent-engineering hub (2024–2026). This is the ge Parent: [LLM Models and APIs](https://llms-explorer.com/tree/llm-models-and-apis/) · 18 facets · 70 facts · page: https://llms-explorer.com/tree/diffusion-generative-media-models/ ## Diffusion & Generative-Media Models — Image / Video / Audio - The model family that generates continuous media (images, video, audio) by learning to reverse a noising process. A model-layer reference under the ai-agent-engineering hub (2024–2026). This is the generation side and is deliberately separate from multimodal-llm-architecture (which is media understanding - vision encoders, VLMs) and from da-35-synthetic-data-generation (which is tabular synthesis). When the task is "make a picture/video/sound," it lives here. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#diffusion-generative-media-models-image-video-audio) ## 1. The denoising-diffusion core (DDPM) - A forward process gradually adds Gaussian noise to data over T steps until it is ~pure noise; a neural network learns the reverse (denoising) process. Training minimizes a simple objective: predict the noise added at a random timestep. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#1-the-denoising-diffusion-core-ddpm) - Parameterizations: predict the noise ε (DDPM), the data x₀, or the v (velocity) target. v-prediction is preferred at high noise/high resolution and for distillation stability. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#1-the-denoising-diffusion-core-ddpm) - Noise schedule: linear, cosine (Nichol & Dhariwal - better for high-res), or the continuous σ-space of EDM. The schedule controls how SNR decays and matters a lot for quality. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#1-the-denoising-diffusion-core-ddpm) - DDPM is the foundation; everything below is either a faster sampler, a better parameterization/space, a better architecture, or a control method on top. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#1-the-denoising-diffusion-core-ddpm) ## 2. The score-based / SDE view - Song & Ermon's score-based generative models unified diffusion under stochastic differential equations: the reverse process integrates the score (∇ₓ log p(x)) learned by the network. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#2-the-score-based-sde-view) - VP-SDE (variance-preserving ≈ DDPM) and VE-SDE (variance-exploding ≈ NCSN). — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#2-the-score-based-sde-view) - The probability-flow ODE: a deterministic ODE with the same marginals as the SDE - enables fast deterministic sampling and exact likelihoods, and is the bridge to flow matching. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#2-the-score-based-sde-view) - EDM (Karras et al.): a cleaner design space - σ-parameterized noise, preconditioning of network in/out, and the Heun sampler; "EDM2" refines training dynamics. EDM's framing is the modern default mental model. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#2-the-score-based-sde-view) ## 3. Latent diffusion (why almost everything runs in latent space) - Pixel-space diffusion is expensive. Latent Diffusion (Rombach et al. → Stable Diffusion) runs the diffusion process in the compressed latent space of a pretrained VAE: encode image → diffuse/denoise the latent → decode. This cut compute ~10–100× and made open text-to-image practical. The VAE's quality (and its KL/VQ regularization) bounds the system's fidelity; the denoiser is conditioned on text via cross-attention to a text encoder (CLIP/T5). — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#3-latent-diffusion-why-almost-everything-runs-in-latent-space) ## 4. Architectures: UNet → Diffusion Transformer - The original denoiser is a UNet (conv encoder–decoder + skip connections + attention at low resolutions). — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#4-architectures-unet-diffusion-transformer) - Diffusion Transformer (DiT) replaces the UNet backbone with a transformer over latent patches - better scaling with parameters/compute, variable-length handling, and reuse of LLM-stack techniques. DiT is now dominant for frontier models. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#4-architectures-unet-diffusion-transformer) - MMDiT (multimodal DiT, SD3) runs separate but interacting streams for text and image tokens; FLUX is a large rectified-flow DiT. The throughline: frontier image/video models are DiTs trained with flow matching (next section). — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#4-architectures-unet-diffusion-transformer) ## 5. Flow matching & rectified flow - A simpler, often-superior alternative training framework that has largely won at the frontier: — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#5-flow-matching-rectified-flow) - Continuous Normalizing Flows / Flow Matching (Lipman et al.): regress a time-dependent velocity field that transports noise to data along a probability path. Conditional FM makes the objective tractable (regress to a per-sample conditional velocity). — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#5-flow-matching-rectified-flow) - Rectified Flow (Liu et al.): learn straight transport paths between noise and data - straighter paths integrate in fewer steps. Reflow iteratively straightens. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#5-flow-matching-rectified-flow) - Stochastic interpolants generalize diffusion + FM under one theory. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#5-flow-matching-rectified-flow) - FM/RF improve sample quality and few-step generation and are the training objective behind SD3 and FLUX. Diff2Flow (CVPR 2025) shows diffusion and FM are close enough to fine-tune a diffusion prior as an FM model by rescaling timesteps - evidence of the paradigms' convergence. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#5-flow-matching-rectified-flow) ## 6. Guidance — classifier & classifier-free - Classifier guidance (Dhariwal & Nichol) steers sampling with a separate classifier's gradient. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#6-guidance-classifier-classifier-free) - Classifier-free guidance (CFG) (Ho & Salimans) is the workhorse: jointly train conditional + unconditional (drop the prompt p% of the time), then at sampling extrapolate ε = ε_uncond + s·(ε_cond − ε_uncond). The guidance scale s trades prompt-adherence vs diversity/fidelity; negative prompts put content in the unconditional branch to push it away. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#6-guidance-classifier-classifier-free) - Costs/caveats: CFG doubles per-step compute (two evals); high s over-saturates - hence CFG rescaling (Lin et al.) and guidance-distillation (§8) to fold CFG into one eval. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#6-guidance-classifier-classifier-free) ## 7. Samplers / schedulers (the steps↔quality knob) - The sampler numerically integrates the reverse ODE/SDE; fewer steps = faster but lower quality: — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#7-samplers-schedulers-the-stepsquality-knob) - DDIM - deterministic, non-Markovian; enables 20–50-step sampling and latent interpolation/inversion. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#7-samplers-schedulers-the-stepsquality-knob) - DPM-Solver / DPM-Solver++ - high-order ODE solvers; good quality at ~10–20 steps (a common default). — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#7-samplers-schedulers-the-stepsquality-knob) - Euler / Euler-a / Heun, UniPC, Karras σ schedule for step spacing. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#7-samplers-schedulers-the-stepsquality-knob) - Rule of thumb: solver choice + step count + schedule jointly set the speed/quality frontier before you reach for distillation. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#7-samplers-schedulers-the-stepsquality-knob) ## 8. Few-step generation & distillation - Pushing from ~20–50 steps down to 1–4: — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#8-few-step-generation-distillation) - Consistency Models (Song et al.) - learn a function mapping any point on a trajectory directly to its origin; sample in 1–2 steps. Latent Consistency Models (LCM) bring this to Stable Diffusion; LCM-LoRA is a plug-in accelerator. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#8-few-step-generation-distillation) - Distillation families: progressive distillation (halve steps repeatedly), guidance distillation (bake CFG into one eval), consistency distillation, adversarial (ADD/SDXL-Turbo, LADD latent ADD) for 1–4-step high-quality, and sCM (score-regularized continuous-time consistency) for large-scale. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#8-few-step-generation-distillation) - Video: TurboDiffusion (ShengShu/Tsinghua, 2025) reports ~100–200× end-to-end speedups via step distillation (rCM) + low-bit SageAttention/sparse-linear attention + W8A8 - enabling near-real-time video. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#8-few-step-generation-distillation) ## 9. Conditioning & control (the six levers) - Customizing/conditioning a frozen or lightly-tuned base model: — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#9-conditioning-control-the-six-levers) - ControlNet - clones the encoder to add spatial conditioning (edges, depth, pose, segmentation) while locking the pretrained backbone. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#9-conditioning-control-the-six-levers) - T2I-Adapter - lighter-weight spatial conditioning. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#9-conditioning-control-the-six-levers) - IP-Adapter - image-prompt (reference-image) style/content conditioning via decoupled cross-attention. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#9-conditioning-control-the-six-levers) - DreamBooth - fine-tune on a few images to bind a subject to a token (with class-preservation loss). — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#9-conditioning-control-the-six-levers) - LoRA - low-rank adapters for cheap style/subject fine-tuning (the dominant community method); composable. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#9-conditioning-control-the-six-levers) - Textual Inversion - learn a new embedding ("a new word") for a concept without touching weights. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#9-conditioning-control-the-six-levers) - (Note: LoRA/DreamBooth for diffusion live here; LoRA/PEFT for LLM text is llm-fine-tuning-peft.) — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#9-conditioning-control-the-six-levers) ## 10. Text-to-video & the video-diffusion stack - Architecture: latent video DiT with spatiotemporal attention (full 3D, or factorized spatial+temporal); a 3D/causal VAE compresses time as well as space. Conditioning and CFG carry over from image diffusion. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#10-text-to-video-the-video-diffusion-stack) - The hard problem is temporal consistency (flicker, identity drift, motion coherence) and cost (sequence length explodes with frames). — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#10-text-to-video-the-video-diffusion-stack) - Landscape (2025–26): closed - Sora/Sora 2, Google Veo 2/3, Kling, Runway Gen-3, Pika, Luma, Minimax/Hailuo. Open - CogVideoX, Mochi-1, HunyuanVideo, Wan (Wan2.x), LTX-Video/LTX-2, Allegro. Diffusers documents the open stack. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#10-text-to-video-the-video-diffusion-stack) - Control for video: WanVideo + ControlNet, image-to-video conditioning, motion LoRAs. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#10-text-to-video-the-video-diffusion-stack) ## 11. Audio, music & other modalities (brief) - Diffusion also drives audio/music generation (e.g. Stable Audio, audio latent diffusion) and 3D/robotics (diffusion policies for manipulation). The same core - denoise in a learned latent, condition via cross-attention, guide with CFG - transfers across modalities. Deep audio/music modeling is out of scope here; this is the connective overview. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#11-audio-music-other-modalities-brief) ## 12. Evaluation & efficiency - Image: FID (Fréchet Inception Distance - distribution match), CLIPScore (prompt alignment), FID-CLIP trade-off curves, increasingly human-preference models (PickScore, ImageReward, HPS). — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#12-evaluation-efficiency) - Video: FVD (Fréchet Video Distance), VBench dimensions (temporal flicker, motion, subject consistency), human eval. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#12-evaluation-efficiency) - Efficiency is its own active survey area (TMLR/TPAMI 2025 efficient-diffusion surveys): architecture, sampler, distillation, and quantization axes - know which axis you're optimizing before reaching for the next trick. — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#12-evaluation-efficiency) ## Sources - Efficient Diffusion Models: A Survey - TMLR 2025 (AIoT-MLSys-Lab); Efficient Diffusion Models - TPAMI 2025 (TsinghuaC3I) — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#sources) - Video Diffusion Models Survey (2025); HuggingFace, State of open video generation models in Diffusers — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#sources) - Diff2Flow: Training Flow Matching Models via Diffusion Model Alignment - CVPR 2025 (CompVis) — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#sources) - On Distillation of Guided Diffusion Models - arXiv 2210.03142; Large-Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency (sCM) - arXiv 2510.08431 — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#sources) - TurboDiffusion: Accelerating Video Diffusion Models by 100–200× - arXiv 2512.16093 (ShengShu / Tsinghua) — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#sources) - Six Ways to Control Style and Content in Diffusion Models - Towards Data Science; Understanding and Training IP-Adapters - Mercity Research — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#sources) - Adaptive Video Distillation: Mitigating Oversaturation and Temporal Collapse - arXiv 2603.21864 — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#sources) - Foundational (pre-cutoff canon): DDPM (Ho 2020), score-SDE (Song 2021), Latent Diffusion/Stable Diffusion (Rombach 2022), CFG (Ho & Salimans 2022), EDM (Karras 2022), Rectified Flow (Liu 2022), Flow Matching (Lipman 2023), Consistency Models (Song 2023), DiT (Peebles & Xie 2023), SD3/MMDiT (Esser 2024) — [source](https://llms-explorer.com/sources/mdb-context-hub/diffusion-generative-media/#sources) ## Where this helps - Choosing between a diffusion model and a GAN or autoregressive model for an image, video, or audio generation task, based on sample diversity and training-stability tradeoffs. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Deciding whether to run diffusion sampling in pixel space or latent space based on your compute budget and target resolution. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Picking a guidance strategy — classifier-free guidance — and its scale to trade off prompt fidelity against sample diversity for a text-to-image or text-to-video pipeline. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Choosing a few-step distillation approach when near-real-time generation latency is needed instead of the 20-50+ step sampling a base diffusion model typically requires. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* ## Project ideas - Implement a basic DDPM-style denoising diffusion model on a small image dataset to build intuition for the forward-noising and reverse-denoising process before using a pretrained model. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Fine-tune a latent diffusion model with a conditioning signal — an edge map, a pose, or a reference image — to build a controllable image-generation tool. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Build a comparison harness that samples the same prompt across different schedulers or samplers (DDIM, DPM-Solver, etc.) at different step counts to visualize the steps-versus-quality tradeoff directly. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Distill a base diffusion model into a few-step, or one-step, variant for a latency-sensitive application, then measure the quality loss against the base model. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* ## Antipatterns - Cranking classifier-free guidance scale up to maximize prompt adherence without checking for the oversaturation and artifacting that high guidance scales tend to introduce. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Choosing a heavily distilled few-step model for a quality-sensitive application without first measuring how much sample quality and diversity it actually gave up versus the base model. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Running full pixel-space diffusion on a high-resolution target when a latent-space model would deliver comparable quality at a fraction of the compute. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Treating video diffusion as "image diffusion but more frames" without budgeting for the temporal-consistency problem — flicker and drift — that image diffusion never had to solve. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* ## Known issues - Diffusion models are computationally expensive at inference relative to a single-forward-pass generator, since even an efficient sampler needs multiple denoising steps. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Classifier-free guidance improves prompt adherence, but at higher guidance scales it tends to produce oversaturated or artifact-heavy outputs, making guidance scale a real quality tradeoff, not a free lunch. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Few-step distillation techniques trade some sample quality and diversity for speed; a distilled model rarely matches its teacher's full-step quality exactly. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Video diffusion models inherit all of image diffusion's compute cost multiplied across frames, plus a temporal-consistency problem — flicker, drift — that image diffusion doesn't have to solve. — [source](https://llms-explorer.com/tree/diffusion-generative-media-models/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* ## Context files - [Diffusion & Generative-Media Models](https://llms-explorer.com/downloads/sources/mdb-context-hub/diffusion-generative-media.md)