SAM 3D: 3Dfy Anything in Images

Give SAM 3D one ordinary photo and a mask that points at one object, and it builds a 3D version of that object: its shape, its surface colors and where it sits in the scene. A Meta team made it, and it received a Best Paper Honorable Mention at CVPR 2026.

SAM 3D (the released model is SAM 3D Objects) rebuilds one object in 3D from a single image and a mask marking it. It outputs shape, texture and layout: the object's rotation, translation and scale in the camera frame. It targets cluttered everyday photos, not clean product shots. A pointmap (per-pixel 3D from depth) is an optional input; the authors estimate one with an off-the-shelf depth model when no sensor data exists.

The authors list four contributions: the model; a human- and model-in-the-loop data pipeline; a training recipe borrowed from language models, from synthetic pretraining to alignment on real images; and SA-3DAO, a benchmark of 1,000 images with artist-made meshes.

Why it matters: accurate 3D labels for natural photos are scarce, so the recipe is a template for data-poor domains. CVPR 2026 lists the paper under Best Paper Honorable Mentions, the conference's recognition and not an EyeCode ranking.

The paper presents SAM 3D as a foundation model for single-image object reconstruction that predicts a complete 3D asset and its scene layout jointly. The headline claim is less a new network than a recipe: adapt the language-model pipeline of large-scale pretraining, mid-training and post-training with human preference alignment to 3D, where real annotated data is orders of magnitude scarcer than text or images.

The four stated contributions are the model, a data engine in which annotators choose among model-generated candidates instead of building meshes from scratch, the staged training framework, and SA-3DAO, whose meshes were produced by professional 3D artists for real photographs and serve as what the authors call an expert human upper bound. Code, checkpoints and a demo are released; the arXiv record shows a revised v2 from June 2026. The CVF open-access page lists the same 22 named authors in the same order as arXiv (arXiv also lists the collective 'SAM 3D Team' first). The official award page places the paper under CVPR 2026 Best Paper Honorable Mentions, and the claims of superiority below are the authors' own.

A photo flattens the world. A mug seen from the side hides its back, and a chair behind a table hides its legs. Many different 3D objects could produce the same picture, so the model has to make a sensible guess. Data for teaching 3D guessing is far rarer than plain image data.

Input: an image I and a mask M selecting one object. Output: shape S, texture T and pose (rotation R, translation t, scale s) in camera coordinates. Because projecting 3D to 2D loses information, the authors model a conditional distribution over answers and train a generative model instead of predicting one fixed answer.

Challenges: real photos have occlusion, clutter and small objects, while accurate 3D labels are scarce and artists need hours per mesh. Prior work, as the paper describes it, mostly trained on isolated synthetic objects and tested on simple benchmarks, and layout methods covered tabletop, street or indoor scenes, often estimating shape and pose in separate steps.

Formally the paper targets $p(S,T,R,t,s \mid I,M)$ and trains a generative approximation $q$. Predicting layout inside the same model, rather than registering a shape afterwards, is deliberate: the full image supplies context that helps decide both what the object is and where it sits. SAM 3D handles occluded objects mainly through data, not through a special architecture.

The bottleneck is supervision. Rendered synthetic assets give shape and texture but leave a domain gap to real photos, and human mesh authoring does not scale. The key observation is that most people cannot create a mesh but can pick which of several meshes best matches an object, so the task is converted from generation to verification. Evaluation is a second gap: the authors argue existing real-image benchmarks are small, indoor and skewed toward furniture, which motivated SA-3DAO, with scenes from parks, ski resorts, flea markets and parades, and artist completion times from about 5 minutes up to over 5 hours per object (median mesh 4,751 vertices).

Instead of paying artists to model everything, let the AI propose several 3D guesses, have people pick the best one, and send only the hardest cases to professional artists. Then retrain on those picks and repeat, like a student who keeps getting graded and improving.

The central idea is a loop between model and annotators. The model and other tools propose candidate 3D shapes for a real image, an annotator picks the best and grades it, accepted results become training data, and rejected candidates become negatives for preference training.

Worked example (our illustration): for a half-hidden chair, about 8 candidates are compared in pairs; the winner is rated and, if good enough, aligned to a point cloud for training. If none is acceptable, a 3D artist may build it.

The second idea is a staged recipe inspired by language models: synthetic pretraining, mid-training on renders pasted into photos, then post-training on real data with SFT and DPO. Late in the project the model produced about 80 percent of the annotated data.

Two ideas interlock. The data engine exploits the asymmetry between generating and verifying: annotators select the best of N=8 candidates from a model ensemble, reject results below a quality threshold, and align accepted meshes to a point cloud. Hard cases that no model can handle go to professional artists and form Art-3DO, so supervision arrives where the model is weakest. The authors describe the loop as an alignment algorithm close to Expert Iteration and RLHF: an ensemble plus human best-of-N amplifies the current policy, and the quality threshold rises over iterations, like the cross-entropy method. Their own summary: dataset generation emerges as a by-product of alignment.

The recipe puts cheap synthetic data at the base and small, human-vetted data at the top. Mid-training adds mask following, occlusion robustness and layout using render-and-paste composites with exact 3D ground truth. Post-training repeats SFT then DPO across several engine iterations, so better models yield better annotations. The paper adds that the flow-matching loss alone sufficed: explicit geometric losses helped in low-data and low-compute settings, but their value shrank as data and compute grew.

Think of two painters. The first sketches a rough block of the object and decides how it is turned and sized in the scene. The second adds fine detail and color. A final step turns the result into a mesh or a cloud of colored blobs.

Inputs: the full image and a masked crop, encoded by DINOv2, plus an optional pointmap. The design extends Trellis's two-stage latent flow matching to predict layout.

• Geometry model: a 1.2 billion parameter flow transformer predicts a coarse 64-cubed grid plus rotation, translation and scale.
• Texture and refinement model: a 600 million parameter sparse flow transformer adds detail and texture on occupied voxels.
• Decoders: output a mesh or 3D Gaussian splats.

Training uses flow matching, then DPO. Distillation cuts geometry sampling from 25 to 4 steps for sub-second output. Pretraining used about 2.7 million meshes rendered from 24 views; mid-training used 61 million render-and-paste samples.

Conditioning: DINOv2 features of a cropped object view and of the full image, each with its mask, give four token sets; an optional scene pointmap comes from a sensor such as LiDAR or from monocular depth. The geometry model models $p(O,R,t,s \mid I,M)$ with $O \in \mathbb{R}^{64^3}$. Shape is denoised in the latent space of a coarser $16^3 \times 8$ representation as in Trellis; layout is denoised directly in parameter space. Modality-specific projections map shape (4,096 tokens) and layout (1 token) to a shared width of 1,024, and the Mixture-of-Transformers uses one transformer for shape tokens and another for layout tokens, tied together by joint self-attention. This lets one modality be fine-tuned alone while rotation stays anchored to shape.

Pretraining and SFT minimize a multi-modal rectified flow-matching loss, $\mathcal{L}=\sum_m \lambda_m \mathbb{E}\,\lVert v^m - v_\theta^m(x^m_\tau, c, \tau)\rVert^2$, with per-modality weights (shape 1.0, rotation 0.1, translation 1.0 and scale 0.1 in the later stages, Table 5). DPO follows Diffusion-DPO adapted to flow matching: for a preferred and a rejected sample, it rewards a lower flow-matching error on the winner than a frozen reference model, relative to the loser. Negatives from non-SAM 3D sources are removed before DPO. Distillation uses shortcut models fine-tuned from trained checkpoints (about 4K iterations, 75 percent flow matching and 25 percent shortcut), with classifier-free guidance distilled in; the paper reports 38 times (1 step) and 10 times (4 steps) faster inference than 25 flow-matching steps. The texture model was not distilled in the final release.

Schedule (Tables 1 and 5): pretraining on Iso-3DO with object-centric crops (2.5 trillion tokens); mid-training on RP-3DO and ProcThor with full-image conditioning, translation and scale added in the second part (about 2.7 trillion tokens); SFT on MITL-3DO then Art-3DO; DPO on 44K preference samples. The last post-training iteration used about 0.5 trillion tokens, and the final SFT run keeps 500K samples above the quality cutoff. The texture model gets analogous stages, a Depth-VAE that back-projects features only to visible voxels, and de-lighted ambient-light renderings so the model learns textures without baked shading. Stage timings: about 10 s to segment an object, 80 s to choose among 6 to 10 candidates and 150 s to align one. Over the project the engine yielded 3.14 million trainable shapes, 1.23 million layout samples, 100K textures and over 7 million pairwise preferences, sourced from 850K object instances in 360K images (appendix A).

The authors compare against other image-to-3D systems on new real-photo tests. Their model gets clearly better shape and layout scores, and in the authors' own preference studies people chose it about five times as often for single objects and six times as often for whole scenes. On clean, isolated object images the gap mostly disappears.

Protocols: shape on SA-3DAO (F-score, voxel IoU, Chamfer, EMD) and ISO3D (ULIP and Uni3D similarity, no ground-truth geometry); layout on SA-3DAO and Aria Digital Twin (3D IoU, rotation error, ADD-S). Baselines include Trellis, Hunyuan3D, Direct3D-S2, TripoSG, Hi3DGen, pose pipelines and MIDI; see the tables below.

Headlines: on SA-3DAO, F1 0.2344 and Chamfer 0.0400 versus 0.1629 and 0.0844 for the best baseline on each; on Aria Digital Twin, joint ADD-S@0.1 is 0.7673 versus 0.0175 for MIDI. Not uniform: on ISO3D the paper says SAM 3D matches or exceeds prior results, and TripoSG has a higher ULIP score.

Human preference: the authors report a 5:1 win rate on real images and 6:1 for scenes over prior methods; per-baseline rates are in a chart we do not quote. Ablation (Table 4): shape F1 climbs from 0.1349 after pretraining to 0.2344, each stage adding something. The appendix shows fair robustness to imperfect masks.

Protocol details. SA-3DAO shape metrics normalize both meshes to [-1,1], align them with ICP and sample 1M points. ISO3D has 101 synthetic in-the-wild images and is scored on 8,192 sampled surface points. Layout metrics are 3D IoU of axis-aligned boxes, ICP-aligned rotation error in degrees, and ADD-S normalized by object diameter, with ADD-S@0.1 counting samples under 10 percent of the diameter. Layout evaluation needs ground-truth depth or pointmaps; elsewhere pointmaps are estimated. Human preference sets: 1,000 SA-1B pairs, 1,000 MetaCLIP pairs with heavy occlusion, 1,000 LVIS images with 10 to 30 objects, and 40 Aria Digital Twin frames, judged by pairwise tournaments with randomized order and anonymized models.

Shape and layout (Tables 2 and 3). SAM 3D leads every SA-3DAO shape column; on ISO3D it has the best Uni3D (0.3707) but not the best ULIP. In layout, giving FoundationPose the SAM 3D shape scores below the joint prediction on both datasets (SA-3DAO 3D IoU 0.2837 versus 0.4254), supporting joint prediction. The text highlights ADD-S@0.1 on Aria Digital Twin rising from 2 percent to 77 percent compared with MIDI. A render-and-compare post-optimization on Aria Digital Twin raised 3D IoU from 0.4837 to 0.5258 and 2D mask IoU from 0.5143 to 0.6487 (Table 9), but only the 554 accepted samples out of 1,027 instances count, because an optimized layout is kept only if its mask IoU improves.

Ablations

• Stages (Table 4): near-monotonic shape gains; vIoU dips slightly after the final DPO (0.2337 to 0.2311) and the DPO texture win rate (66.4) is below the SFT one (66.9).
• Knockouts (Table 7): without MITL-3DO F1 0.2211, without Art-3DO 0.2027, and the row labelled without DPO on MITL-3DO 0.2156. The last two rows repeat Table 4's SFT(MITL) and DPO(MITL) values exactly, so the labelling of knockouts versus cumulative stages is not fully clear.
• Rotation (Table 10, 216 held-out Objaverse samples): normalized 6D rotation gives Chamfer 0.0049 and ICP-Rot 14.5946 against 0.0061 and 17.9585 for quaternions; plain 6D is better than quaternions on rotation (15.5399) but worse on Chamfer (0.0074).
• Pointmap: with and without it, shape was each preferred 48 percent of the time on LVIS.
• Depth-VAE (Table 11, GSO): PSNR 30.65 to 30.87, then 31.60 with more data; LPIPS 0.04776 to 0.04093.
• Best-of-N recovery (Table 12): fine-tuning with 50-sample recovered data versus N=2 lifts F1 on the tail holdout 0.39 to 0.41 and on Epic Kitchens 0.30 to 0.32 but leaves SA-3DAO at 0.26. Recovery yield for formerly failed samples rose from 0 to 86.8 percent.

Texture (Table 8). With SAM 3D shape given to every method, SAM 3D wins human preference rates between 63.8 and 89.1 percent against Trellis, Hunyuan3D 2.0 and 2.1, and Unitex across four sets. The printed Trellis and Hunyuan3D-2.1 rows are identical on three of four columns (87.0, 86.2, 89.1) in both the PDF and HTML, which looks like a copy error. Reward models for data recovery agreed with human binary choices 68.9 percent (a VLM) and about 65 percent (DPO implicit reward), against under 75 percent between two humans.

Interpretation. Gains come from combining real-data post-training and the data engine, and the texture ablation (Figure 17) says lighting augmentation matters most, RP-3DO data are critical, and post-training data and DPO give significant gains. The benchmark, preference sets and baselines are chosen by the authors, so independent replication on SA-3DAO, now on a public leaderboard, is the real test.

The 3D shape is fairly coarse, so thin parts, hands or faces can lose detail. Objects are guessed one at a time, so the model does not check whether two objects overlap or float, and the texture ignores how the object is turned.

Reported by the authors

Appendix F: geometry is a coarse 64-cubed grid, so thin parts, hands and faces may distort; objects are reconstructed one at a time without contact or stability reasoning; texture ignores pose, so symmetric objects can come out rotated wrongly. Elsewhere they note compounding errors from automatic masks and reward models agreeing with humans only about 65 to 69 percent.

EyeCode analysis (editorial interpretation)

The benchmark, preference studies and baselines are the authors' own, and Table 8 has repeated cells. Training needed heavy annotation and GPU resources (512 A100s for geometry pretraining), so full reproduction is hard even with released checkpoints.

Reported by the authors

Appendix F: resolution is bounded by the 64-cubed occupancy grid and up to 32 splats per voxel; suggested remedies are higher output resolution through architecture changes, a super-resolution model, parts-based generation or an implicit representation. Layout is predicted per object, so physical interactions are not modeled; multi-object prediction with suitable losses is left as a next step. Texture lacks pose knowledge, hurting rotational symmetry. Related notes: with imperfect masks the model shows fair robustness (Figure 20) but errors can compound; reward models are imperfect proxies, and the authors attribute part of the disagreement to near-identical meshes or meshes better in some regions and worse in others.

EyeCode analysis (editorial interpretation)

The reported gains lean on in-house evaluation sets whose protocol (artist meshes, ICP alignment, pairwise human judging) the authors designed, so absolute numbers are best compared across methods within their tables only. ISO3D, taken from 3D Arena, is the one shape benchmark the authors did not build, and there the model is on par, not ahead. The cumulative-stage design cannot isolate whether Art-3DO helps through data quality or through its aesthetic alignment effect (the paper says it suppresses floaters, bottomless meshes and missing symmetry). No per-category error breakdown is reported, so how performance varies by object type stays open. The duplicated Table 8 cells and the unclear Table 7 labels should be checked against the released leaderboard and code.

A strong 3D model can come from a good data loop, not only from a clever network: let the model propose, people choose, and repeat.

• Verification is cheaper than generation: picking among meshes scales labels to millions of samples.
• Each training stage adds measurable shape quality on SA-3DAO.
• On clean ISO3D images the model is on par, so the strongest evidence is for cluttered real photos.
• Limits: coarse geometry, one object at a time, no pose-aware texture.

Suggested directions (EyeCode suggestions, not claims from the paper)

• Run the released model on your own cluttered photos.
• Compare against the public SA-3DAO leaderboard.

• Method lesson: turn a task people cannot do (modeling meshes) into one they can (ranking candidates), and keep rejected candidates for DPO.
• Training lesson: synthetic pretraining generalizes when followed by enough real post-training; small artist data (Art-3DO) gave a large marginal gain in the cumulative table.
• Design lesson: joint shape and layout in one Mixture-of-Transformers outperformed separate shape plus pose pipelines in the layout comparison.
• Loss lesson: flow matching without explicit geometric losses sufficed at scale.
• Reading lesson: separate benchmark metrics from human preference studies and note which datasets are the authors' own.

Suggested directions (EyeCode suggestions, not claims from the paper)

• Study joint multi-object prediction with contact and stability losses, which the authors name as future work.
• Test higher-resolution or parts-based outputs against the 64-cubed limit.
• Measure how reward-model best-of-N changes data yield across object categories.