From ε to Worlds
The mathematics of world models: diffusion, flow matching, Dreamer and JEPA

From ε to Worlds

The mathematics and the ideas, built in order, so that the papers read easily

How this book works

You want to read a particular shelf of papers: DiT, Wan, LTX-Video, Diffusion Forcing, DMD2, CausVid, Self Forcing, Causal Forcing, GameNGen, DIAMOND, Matrix-Game, LongLive, WorldMem, StreamDiffusionV2, and after them Dreamer and JEPA. They share one body of mathematics and they all assume you have it. This book builds that mathematics in the order it depends on itself, and stops at each step to show you where the papers use it.

It assumes a strong programmer who has met calculus, linear algebra and probability but does not use them every day. Every idea arrives in the same order: the easy case, a number, a picture of samples, and only then the definition and the proof. Every section ends with questions to check yourself, and each chapter with a box, In the papers, that translates the chapter into the notation of the papers, because half the difficulty of this literature is that four traditions write the same thing four ways.

The title is the whole story in two symbols. ε is the noise every sample in the first half of the book starts from; the worlds are where it ends: generated video you can act in, and models that imagine the consequences of an action.

Contents

A chapter is linked here once it is published; the rest of the plan is listed so you can see where the book goes.

Part I: The mathematics you need

  1. Probability and expectation: densities; joint, marginal, conditional, Bayes; mixtures and posteriors; expectation, variance, covariance; conditional expectation as the best predictor; Monte Carlo, the law of large numbers, the central limit theorem, mini-batches
  2. The Gaussian: the normalizing constant; change of variables; why variances add; covariance matrices; conditioning (precisions add); the geometry of high-dimensional noise; the score
  3. Comparing distributions: likelihood and maximum likelihood; KL divergence; Jensen's inequality and why KL is non-negative; forward vs. reverse KL (mode-covering vs. mode-seeking); the KL between two Gaussians
  4. Moving a point: vector fields, ODEs, flows, Euler's method
  5. Moving a whole distribution: probability paths, push-forward, the continuity equation
  6. Motion with randomness: Brownian motion, SDEs, Euler-Maruyama, Fokker-Planck

Part II: Training a generator

  1. Flow matching: conditional paths, the marginalization trick, the CFM loss; the straight-line (rectified) path
  2. Scores and diffusion: the score function, denoising score matching, Tweedie's formula; DDPM as a special case; the ideal denoiser of a finite training set (a softmax-weighted average that memorizes) and why real networks generalize
  3. One network, four outputs: ε, x₀, v and score, and how to convert; noise schedules, SNR, timestep shift; EDM's σ-parameterization and preconditioning
  4. Sampling: the probability-flow ODE, SDE samplers, Euler/Heun/DPM-style solvers, steps vs. quality
  5. Guidance: conditioning, classifier-free guidance, what the guidance scale really does

Part III: The machines

  1. Latent space: VAEs, the KL term, compression ratios; causal 3D video VAEs (LTX-Video, Wan)
  2. Diffusion Transformers: patchify, adaLN-zero, attention, RoPE in space and time, text conditioning, scaling (DiT, Wan)

Part IV: Making it fast

  1. Distillation I, following the ODE: flow maps, consistency models, ODE-regression initialization
  2. Distillation II, matching distributions: reverse KL, "the gradient is a difference of two scores", the fake-score network, the GAN term (DMD2)

Part V: Time, causality, interaction

  1. Video as a sequence: bidirectional vs. causal attention, teacher forcing, exposure bias; per-token noise levels (Diffusion Forcing)
  2. From slow to real-time: CausVid -> Self Forcing -> Causal Forcing / Causal Forcing++: the train-test gap, KV caches, rollouts, the injectivity argument
  3. World models: action conditioning, drift and noise augmentation (GameNGen, DIAMOND, Matrix-Game 2.0)
  4. Long horizons and memory: KV re-caching, frame sinks, memory banks (LongLive, WorldMem)
  5. Systems: streaming, latency budgets, batching across denoising steps (StreamDiffusionV2)

Part VI: Acting in a world

  1. Decisions and returns: Markov decision processes, returns and discounting, value functions, the Bellman equation (DIAMOND's agent, Dreamer)
  2. Learning to act: policy gradients, actor-critic, λ-returns, entropy regularization; training a policy inside a learned model ("in imagination")

Part VII: Latent world models

  1. Variational inference in time: latent states, the ELBO for sequences, KL balancing and free bits (Dreamer)
  2. Dreamer: the recurrent state-space model, discrete latents and straight-through gradients, symlog and return normalization, imagination rollouts (DreamerV3)

Part VIII: Predicting in representation space

  1. Why not predict pixels?: the energy-based view, prediction in embedding space, representation collapse and the ways to prevent it (stop-gradient, EMA targets, variance-covariance penalties)
  2. JEPA: masked prediction in latent space for images and video (I-JEPA, V-JEPA), planning with a JEPA world model; how the three families compare