Navigation World Models

Imagine a robot that can daydream. Before it moves, it pictures what it would see after turning left or walking forward, and then picks the route whose daydream looks most like the place it wants to reach. Navigation World Models (NWM) is a video-generating model that gives a navigating agent exactly this ability. It also lets the agent follow a new rule, such as 'go straight first', without being taught again.

Navigation World Models (NWM) is a controllable video-prediction model for first-person navigation. From a few past camera frames, a movement command (how far to move and how much to turn) and a time offset, it generates the next frame. Chaining predictions gives a video of an imagined trip.

The authors use this learned simulator for planning. They imagine many candidate trajectories, compare the final imagined frame of each with a goal image, and keep the best one. The model can also re-rank the proposals of an existing policy, NoMaD, so it can be added to a system that already works.

Main contributions: a Conditional Diffusion Transformer (CDiT) whose cost grows linearly with the number of context frames; training on several robot datasets plus unlabeled Ego4D video; and planning with and without constraints. Why it matters: planning by simulation lets a navigation system spend more compute on hard cases and accept new rules without retraining. The paper received a Best Paper Honorable Mention at CVPR 2025, according to the official awards page.

NWM is a conditional diffusion transformer (CDiT-XL, about 1B parameters in the default setting) trained as an action-conditioned world model on egocentric robot video. The central claim is that a good video predictor is more than a visualisation tool: at test time it works as a search engine that can replace, or re-rank, a feed-forward goal-conditioned policy.

The paper evaluates three roles: predicting frames over horizons up to 16 seconds, planning from scratch with a sampling-based optimiser, and ranking trajectories proposed by NoMaD. It also studies how unlabeled Ego4D video affects unseen environments, and how constraints are imposed at planning time. The authors describe NWM as conceptually close to diffusion world models for offline reinforcement learning (DIAMOND, GameNGen) but trained across many environments and embodiments. Recognition: Best Paper Honorable Mention at CVPR 2025; the interpretation below is EyeCode's own.

Most navigation software learns a fixed reflex: look at the camera image and the goal, output the next move. It cannot weigh alternatives or easily accept a new rule. The question here is whether an agent can learn how its view changes when it moves, and then think ahead using that knowledge.

Typical visual-navigation systems learn a policy, a network that maps the current observation and a goal image directly to actions. The authors point out two limits: after training, such policies are 'hard-coded' and new constraints cannot easily be added, and they cannot dynamically allocate more computation to hard problems.

The question asked: can an agent learn a world model of how its view changes under its own movement and plan inside it at test time? Formally, the model receives past latent observations and an action (translation, yaw, time shift) and must output the next latent observation. It has to work across several robots and environments, stay useful over long horizons, and be fast enough to try many candidate plans.

The setting is goal-conditioned visual navigation from egocentric RGB: given context frames and a goal image, find an action sequence (planar translation and yaw) that reaches the goal. Policies such as GNM and NoMaD solve it by direct regression or a diffusion policy trained by behaviour cloning. The paper formulates a stochastic mapping from past latents and an action to the next latent, with the encoder a pretrained VAE, so predictions can be decoded back to pixels.

Challenges: existing diffusion world models (DIAMOND, GameNGen) target a single game; here one model must cover several platforms with different cameras, errors compound over long rollouts, actions and elapsed time can be entangled in the data, and plans must be evaluated cheaply. The authors also want generalisation to GO Stanford, which is held out of training, and the ability to learn from videos without any action labels.

Think of chess: instead of one instinctive move, you try several in your head and see how each plays out. The model generates a short video for each possible route, and the route whose final picture is closest to the goal photo wins. For example, to reach a door, it imagines three paths and keeps the one that ends facing the door.

The key idea is to treat navigation planning as a search over imagined videos. A conditional diffusion transformer predicts the next frame from past frames and an action. Because it is generative, we can sample many action sequences, simulate each, and score how close the final frame is to the goal image.

Three ingredients make this work. First, CDiT: self-attention only on the frame being generated and cross-attention to past frames, which keeps the cost low. Second, a time-shift input: the action also says how many seconds ahead to predict, which lets the model learn from unlabeled videos where only elapsed time is known. Third, constraints: a rule such as 'move forward first' is applied by restricting the candidate actions, with no retraining. Compared with a policy such as NoMaD, which outputs actions directly, NWM can compare alternatives before acting.

Three design choices carry the paper. First, the CDiT block separates denoising of the target frame (self-attention over its own tokens) from conditioning on history (a cross-attention layer to past-frame tokens with a skip connection). The paper states the attention cost as $O(m^2 n^2 d)$ for a DiT that feeds all frames in (m frames, n tokens per frame, width d) versus $O(m n^2 d)$ for the CDiT cross-attention.

Second, the action is a = (u, phi, k): planar translation, yaw and time shift k up to plus or minus 16 seconds. Actions between two frames are composed by summing translations and yaws (Eq. 2). Sampling several goals per state is meant to create natural counterfactuals, so the model cannot rely on time alone. Third, planning is energy minimisation: negative perceptual similarity to the goal plus indicator penalties for invalid actions or unsafe states, optimised with the Cross-Entropy Method inside a Model Predictive Control view.

Each camera frame is squeezed into a compact code. The model repeatedly cleans up noise to produce the next code, guided by the past frames and the chosen movement. To plan, it tries about a hundred simple paths, imagines the result of each, and keeps the one that looks closest to the goal.

Each frame is compressed by the Stable Diffusion VAE into latent tokens. An action holds a 2D translation, a yaw angle and a time shift; each scalar is embedded with sine-cosine features and a small MLP, and the action, time-shift and diffusion-timestep embeddings are summed and injected through adaptive layer normalisation. In CDiT, self-attention covers only the frame being denoised, while cross-attention reads past-frame tokens. Training minimises the standard mean-squared denoising loss, plus a variational lower-bound term for the predicted covariance, following DiT. The default model has about 1B parameters and four context frames.

Planning uses Model Predictive Control with the Cross-Entropy Method (CEM). It samples 120 candidate trajectories of eight steps (0.25 s each, so 2 seconds), simulates each three times, and scores the final frame against the goal with LPIPS. In ranking mode, NoMaD proposes 16 or 32 trajectories and NWM keeps the one whose final frame is closest to the goal. Constraints are enforced by zeroing parts of the action sequence.

Model and training

Inputs are VAE latents of the context frames and the noisy target latent. In each block the target tokens attend to themselves, then cross-attend to past-frame keys and values. Scalars (translation, yaw, time shift and diffusion timestep) go through sine-cosine features and a two-layer MLP, are concatenated into a vector, summed into one conditioning vector and passed to AdaLN, which produces scale and shift for the layer norms and attention outputs. For unlabeled data the explicit action is simply left out of that sum. Noise schedule and hyperparameters follow DiT. The default CDiT-XL uses batch size 1024 with four goals per state (4096 total), AdamW at 8e-5, and 8 H100 machines with 8 GPUs each; reported numbers average five samples. Goals are drawn within a window of plus or minus 16 seconds.

Planning

Standalone planning assumes a straight-line trajectory and optimises only its endpoint (forward offset, sideways offset, yaw), expanded into eight equal delta steps with the yaw applied at the last step. CEM runs one iteration with 120 samples; each is evaluated three times because NWM is stochastic. Initial means are zero except the forward component, whose mean and variance are set per dataset (for example -0.1 and 0.02 on RECON). In ranking mode NoMaD is conditioned on past frames and the goal, 16 or 32 length-8 trajectories are sampled, and the lowest-LPIPS final prediction is selected. Backward motion is filtered from the data, following NoMaD. Constraint examples: forward-first moves forward for 5 steps then turns for 3; left-right-first turns for 3 steps before moving forward.

On four robot datasets the model produced more realistic long videos than an earlier world model. When it planned by itself on one benchmark it got closer to the goal than two popular navigation policies, but on another benchmark it did not win. Extra walking videos from the internet helped in a brand-new place and slightly hurt in several familiar ones.

Data: four robot datasets with actions (SCAND, TartanDrive, RECON, HuRoN), plus unlabeled Ego4D video for training and GO Stanford for unseen-environment tests. Metrics: ATE and RPE for planned trajectories; LPIPS, DreamSim and PSNR for frame prediction; FID and FVD for generation quality. Baselines: DIAMOND (a UNet diffusion world model), GNM and NoMaD (navigation policies).

For 16-second videos at 4 FPS, NWM has far lower FVD than DIAMOND on every dataset, for example 200.969 versus 762.734 on RECON (second table below). In planning, standalone NWM reaches RECON ATE 1.13 and RPE 0.35, versus 1.87 and 0.73 for GNM and 1.95 and 0.53 for NoMaD (first table below, lower is better). It also leads on SCAND and, narrowly, TartanDrive, but not on HuRoN, where NWM + NoMaD (x32) is best at 3.68. Ablations on RECON (third table): four goals instead of one, four context frames instead of one, and action plus time conditioning all help; time alone fails. Training with extra Ego4D video improved GO Stanford slightly (LPIPS 0.658 to 0.652) but worsened RECON (0.295 to 0.368). The authors also show qualitative rollouts, including predictions from a single image in unseen places, and a failure case.

Protocol. Planning is scored on 100 examples per dataset that an always-forward heuristic predicts worst, so the evaluation sets are deliberately hard; the Forward row in the first table is that heuristic, and on TartanDrive, which is dominated by forward motion, the paper notes that results sit close to this baseline. Time-prediction sets use 500 random examples. Ablation numbers are single-step predictions 4 seconds ahead on RECON. Tables print means with standard deviations over five samples; EyeCode shows the means only.

Ablations (third table). Going from one to four goals lowers LPIPS from 0.312 to 0.296, and from one to four context frames from 0.304 to 0.296; the absolute gaps are small but outside the printed standard deviations. Time-only conditioning collapses (0.760), action-only gives 0.318 and action plus time 0.295, which supports the claim that both signals help. CDiT vs DiT (Figure 5, plotted as LPIPS against TFLOPs; we could not read the exact values from the text): the paper says CDiT keeps improving up to 1B parameters at under 2x FLOPs and that CDiT-L is 4x faster than DiT-XL and better; the introduction states 4x fewer FLOPs. Those are different comparisons, not necessarily a contradiction.

Video quality. Figure 4 compares NWM with DIAMOND at 1 and 4 FPS over 16 seconds: the 1 FPS variant is better at first, but after about 8 seconds errors accumulate and the 4 FPS variant is better. Constraints (Table 3): relative to unconstrained planning, the final-position and yaw differences are +0.36/+0.61 for forward-first, -0.03/+0.20 for left-right-first and +0.08/+0.22 for straight-then-forward; the authors read this as minor drops. Unseen data (Tables 4, 5, 9): on GO Stanford LPIPS goes 0.658 to 0.652 with Ego4D, DreamSim 0.478 to 0.464, and test-time adaptation (2k fine-tuning steps) reaches 0.650. On known datasets Ego4D hurt HuRoN (0.250 to 0.377) and TartanDrive (0.414 to 0.430) and helped only SCAND (0.403 to 0.398). Runtime (Table 8, one RTX 6000 Ada, 32 four-second trajectories): 30.3 s, 14.7 s with time skip (8 states instead of 16), 0.4 s after distilling denoising from 250 to 6 steps; the 4-bit figure (0.1 s) is an estimate, not a measurement.

Over long imagined rollouts in unfamiliar places the model slowly forgets where it is and slides toward scenes it saw in training. It also struggles with moving things like pedestrians, it only understands flat 2D movement, and planning is slow without speed-up tricks.

Reported by the authors

The paper has a short limitations section. Out of distribution, the model tends to lose context slowly and produce frames that resemble the training data (mode collapse). It struggles with temporal dynamics such as pedestrian motion, though not always. The authors expect both problems to ease with longer context and more training data. Actions have three degrees of freedom; extending to 6 DoF or to robot-arm joints is left for future work. The authors also say inference speed is a key bottleneck for real robots, and in the discussion they admit they are not sure what representations make NWM work, since it uses no explicit map.

EyeCode analysis (editorial interpretation)

The results are mixed rather than uniformly better: standalone planning loses to ranking NoMaD proposals on HuRoN, and unlabeled Ego4D data trades in-domain accuracy for unseen-environment accuracy. Evaluation uses recorded datasets and deliberately hard 100-example subsets; in the text we read we did not see a real-robot deployment. The paper's tables also disagree slightly with each other (see the caption of the first table), which is worth knowing when reproducing numbers.

Reported by the authors

Section 5 lists mode collapse on out-of-distribution data (shown in Figure 10), weak simulation of independently moving agents such as pedestrians, and 3-DoF actions only. The appendix states that deployment is bottlenecked by inference speed, and offers time skip, distillation (250 to 6 denoising steps, with minor visual quality loss) and an untested 4-bit quantisation estimate. The discussion notes that it is unclear which representations enable navigation, and speculates that next-frame prediction from an egocentric view may lead to allocentric representations; this is not tested.

EyeCode analysis (editorial interpretation)

The planner is simple: a straight-line endpoint, one CEM iteration and two-second horizons, so conclusions about long-horizon planning are not supported. LPIPS similarity to the goal is a proxy that need not reflect reachability. Ablation gaps are small in absolute LPIPS, and Ego4D effects have different signs across datasets, so they should be reported per dataset. The two NoMaD RECON ATEs (1.93 in Table 2, 1.95 in Table 7), and similar small differences for the NWM + NoMaD rows, show the paper is not fully internally consistent; the FLOPs wording (4x fewer in the introduction, under 2x in Section 4.2, 4x faster for CDiT-L vs DiT-XL) also needs care when quoted.

A model that can imagine the outcome of its moves lets a robot plan by trial in its head, follow new rules, and learn from videos without movement labels.

• An action-conditioned video model can serve as a navigation planner, not only a visualiser.
• CDiT conditions on past frames through cross-attention, so cost grows linearly with context length.
• A time-shift input lets unlabeled video join training.
• Standalone planning beat GNM and NoMaD on RECON and SCAND ATE, but not on HuRoN.

Suggested directions (EyeCode suggestions, not claims from the paper)

• Try planners richer than a one-iteration straight-line CEM.
• Report Ego4D mixing per dataset to find when unlabeled data helps.

• Test-time search over a learned simulator is a viable alternative to a direct policy, with constraints expressed as action restrictions rather than retraining.
• Linear cost in context length is what makes scaling the conditioning to about 1B parameters practical.
• Mixed-domain data is a trade-off: it helped GO Stanford and SCAND but hurt RECON, HuRoN and TartanDrive (Tables 4 and 5).
• Evaluation sets matter: results on 100 hardest-for-forward examples may not transfer to typical trajectories.

Suggested directions (EyeCode suggestions, not claims from the paper)

• Replace the LPIPS goal score with a learned reachability or distance measure and compare rankings.
• Run the planner for several CEM iterations and longer horizons to see where simulation error starts to dominate.
• Measure drift in out-of-distribution rollouts as a function of context length to test the authors' longer-context hypothesis.