From ε to Worlds
How this book works
You want to read a particular shelf of papers: DiT, Wan, LTX-Video, Diffusion Forcing, DMD2, CausVid, Self Forcing, Causal Forcing, GameNGen, DIAMOND, Matrix-Game, LongLive, WorldMem, StreamDiffusionV2, and after them Dreamer and JEPA. They share one body of mathematics and they all assume you have it. This book builds that mathematics in the order it depends on itself, and stops at each step to show you where the papers use it.
It assumes a strong programmer who has met calculus, linear algebra and probability but does not use them every day. Every idea arrives in the same order: the easy case, a number, a picture of samples, and only then the definition and the proof. Every section ends with questions to check yourself, and each chapter with a box, In the papers, that translates the chapter into the notation of the papers, because half the difficulty of this literature is that four traditions write the same thing four ways.
The title is the whole story in two symbols. ε is the noise every sample in the first half of the book starts from; the worlds are where it ends: generated video you can act in, and models that imagine the consequences of an action.
What this replaces
Nothing in this book is secret, and none of it is new mathematics. All of it is written down somewhere else — in about fourteen books, on the order of eight thousand pages, of which perhaps a third bears on this subject. Here is that shelf, set against the chapters it would cover. Most of the books are legally free.
- Chapters 1–2 · probability, the Gaussian. Blitzstein and Hwang, Introduction to Probability, or Wasserman’s All of Statistics; the multivariate Gaussian in Bishop’s Pattern Recognition and Machine Learning.
- Chapter 3 · KL divergence and Jensen’s inequality. Cover and Thomas, Elements of Information Theory, or MacKay’s Information Theory, Inference, and Learning Algorithms.
- Chapter 4 · vector fields, flows, ODE solvers. Strogatz, Nonlinear Dynamics and Chaos, with Hairer, Nørsett and Wanner for the solvers.
- Chapter 5 · probability paths and the continuity equation. Santambrogio, Optimal Transport for Applied Mathematicians, or Peyré and Cuturi’s Computational Optimal Transport. Neither is pitched at this reader; the continuity equation is more often picked up from fluid dynamics, at several times the cost of the chapter.
- Chapter 6 · Brownian motion, SDEs, Fokker–Planck. Särkkä and Solin, Applied Stochastic Differential Equations; Øksendal for the rigorous version.
- Chapter 8 · diffusion. One chapter each in Prince’s Understanding Deep Learning, Bishop and Bishop’s Deep Learning: Foundations and Concepts, and Murphy’s Probabilistic Machine Learning: Advanced Topics.
- Chapter 12 · variational autoencoders. Kingma and Welling’s monograph, with Murphy again for the variational machinery.
- Chapters 21–23 · decisions, policies, inference in time. Sutton and Barto, Reinforcement Learning, and Särkkä’s Bayesian Filtering and Smoothing.
- Underneath all of it. Strang or Axler for linear algebra; Hubbard and Hubbard for Jacobians and the change of variables; Goodfellow, Bengio and Courville, or Prince again, for deep learning.
That shelf carries you to about Chapter 13. Twelve of the twenty-six chapters here have no book at all — 9, 10, 11, 14, 15, 16 through 20, 24 and 26 — and two more, flow matching and diffusion transformers, have only a guide and a set of course notes. For those, there is the literature: unindexed, written for people who already hold the mathematics, and using four incompatible notations for the same object. That is the half of the field this book exists for, and it is the half that is still moving.
So this is not a summary of fourteen books. It is a path through them — the parts this subject actually uses, in the order they depend on one another, with the notations reconciled once and the derivations done rather than cited — and then it keeps going after the books stop.
Contents
A chapter is linked here once it is published; the rest of the plan is listed so you can see where the book goes.
Part I: The mathematics you need
- Probability and expectation: densities; joint, marginal, conditional, Bayes; mixtures and posteriors; expectation, variance, covariance; conditional expectation as the best predictor; Monte Carlo, the law of large numbers, the central limit theorem, mini-batches
- The Gaussian: the normalizing constant; change of variables; why variances add; covariance matrices; conditioning (precisions add); the geometry of high-dimensional noise; the score
- Comparing distributions: likelihood and maximum likelihood; KL divergence; Jensen's inequality and why KL is non-negative; forward vs. reverse KL (mode-covering vs. mode-seeking); the KL between two Gaussians
- Moving a point: vector fields, ODEs, flows, Euler's method
- Moving a whole distribution: probability paths, push-forward, the continuity equation
- Motion with randomness: Brownian motion, SDEs, Euler-Maruyama, Fokker-Planck
Part II: Training a generator
- Flow matching: conditional paths, the marginalization trick, the CFM loss; the straight-line (rectified) path
- Scores and diffusion: the score function, denoising score matching, Tweedie's formula; DDPM as a special case; the ideal denoiser of a finite training set (a softmax-weighted average that memorizes) and why real networks generalize
- One network, four outputs: ε, x₀, v and score, and how to convert; noise schedules, SNR, timestep shift; EDM's σ-parameterization and preconditioning
- Sampling: the probability-flow ODE, SDE samplers, Euler/Heun/DPM-style solvers, steps vs. quality
- Guidance: conditioning, classifier-free guidance, what the guidance scale really does
Part III: The machines
- Latent space: VAEs, the KL term, compression ratios; causal 3D video VAEs (LTX-Video, Wan)
- Diffusion Transformers: patchify, adaLN-zero, attention, RoPE in space and time, text conditioning, scaling (DiT, Wan)
Part IV: Making it fast
- Distillation I, following the ODE: flow maps, consistency models, ODE-regression initialization
- Distillation II, matching distributions: reverse KL, "the gradient is a difference of two scores", the fake-score network, the GAN term (DMD2)
Part V: Time, causality, interaction
- Video as a sequence: bidirectional vs. causal attention, teacher forcing, exposure bias; per-token noise levels (Diffusion Forcing)
- From slow to real-time: CausVid -> Self Forcing -> Causal Forcing / Causal Forcing++: the train-test gap, KV caches, rollouts, the injectivity argument
- World models: action conditioning, drift and noise augmentation (GameNGen, DIAMOND, Matrix-Game 2.0)
- Long horizons and memory: KV re-caching, frame sinks, memory banks (LongLive, WorldMem)
- Systems: streaming, latency budgets, batching across denoising steps (StreamDiffusionV2)
Part VI: Acting in a world
- Decisions and returns: Markov decision processes, returns and discounting, value functions, the Bellman equation (DIAMOND's agent, Dreamer)
- Learning to act: policy gradients, actor-critic, λ-returns, entropy regularization; training a policy inside a learned model ("in imagination")
Part VII: Latent world models
- Variational inference in time: latent states, the ELBO for sequences, KL balancing and free bits (Dreamer)
- Dreamer: the recurrent state-space model, discrete latents and straight-through gradients, symlog and return normalization, imagination rollouts (DreamerV3)
Part VIII: Predicting in representation space
- Why not predict pixels?: the energy-based view, prediction in embedding space, representation collapse and the ways to prevent it (stop-gradient, EMA targets, variance-covariance penalties)
- JEPA: masked prediction in latent space for images and video (I-JEPA, V-JEPA), planning with a JEPA world model; how the three families compare