Text-to-3D: the DreamFusion paper, using 2D diffusion

The DreamFusion paper uses a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis.

Verified

Definition

The DreamFusion paper uses a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. The paper introduces a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. The paper says its approach requires no 3D training data and no modifications to the image diffusion model, and that DreamFusion does not require 3D or multi-view training data.

In the paper's introduction, Neural Radiance Fields, or NeRF, are described as an approach towards inverse rendering in which a volumetric raytracer is combined with a neural mapping from spatial coordinates to color and volumetric density. The paper says the resulting Score Distillation Sampling (SDS) method enables sampling via optimization in differentiable image parameterizations, and that by combining SDS with a NeRF variant tailored to this 3D generation task, DreamFusion generates high-fidelity coherent 3D objects and scenes for a diverse set of user-provided text prompts.

At a glance

The paper says that, to synthesize a scene from text, it initializes a NeRF-like model with random weights, then repeatedly renders views of that NeRF from random camera positions and angles, using these renderings as the input to the score distillation loss function that wraps around Imagen. The paper describes what each iteration of DreamFusion optimization performs.

Diagram: Initialize a NeRF-like model with random weights. > Randomly sample a camera and light. > Render an image of the NeRF from that camera and shade with the light. > Compute gradients of the SDS loss with respect to the NeRF parameters. > Update the NeRF parameters using an optimizer.1Initialize a NeRF-like model with randomweights.2Randomly sample a camera and light.3Render an image of the NeRF from that camera andshade with the light.4Compute gradients of the SDS loss with respectto the NeRF parameters.5Update the NeRF parameters using an optimizer.

How it works

The paper describes Dream Fields as using frozen image-text joint embedding models from CLIP and an optimization-based approach to train NeRFs, and says it adopts a similar approach to Dream Fields but replaces CLIP with a loss derived from distillation of a 2D diffusion model. In the paper, the scene is represented by a Neural Radiance Field that is randomly initialized and trained from scratch for each caption. The paper renders the NeRF from a random camera, using normals computed from gradients of the density to shade the scene with a random lighting direction.

For the diffusion model, the paper uses the Imagen model, which has been trained to synthesize images from text. The paper says its model is built upon mip-NeRF 360, which is an improved version of NeRF that reduces aliasing. In the paper's model, the MLP parameterizes the color of the surface itself, which is then lit by an illumination that the authors control, a process commonly referred to as shading.

The paper found it beneficial to append view-dependent text to the provided input text based on the location of the randomly sampled camera. The paper randomly chooses between the illuminated color render, a textureless render, and a rendering of the albedo without shading. The paper's 3D scenes are optimized on a TPUv4 machine with 4 chips, parameters are optimized using the Distributed Shampoo optimizer, and compute time is split evenly between rendering the NeRF and evaluating the diffusion model.

Example

The paper shows that DreamFusion generates 3D objects from a natural language caption such as "a DSLR photo of a peacock on a surfboard". In the paper's ablation study, the base method without view-dependent prompts results in a multi-faced dog with flat geometry. In the same ablation, adding view-dependent prompts improves geometry, but the surfaces are highly non-smooth and result in poor shaded renders.

The paper evaluates with the 153 prompts from the object-centric COCO validation subset of Dream Fields, using R-Precision, the accuracy with which CLIP retrieves the correct caption among a set of distractors given a rendering of the scene. The paper reports that in its CLIP R-Precision evaluation, DreamFusion outperforms both baselines, Dream Fields and CLIP-Mesh, on color images and approaches the performance of ground truth images, and it says these two baselines have an unfair advantage because they use CLIP during training. The paper reports that when evaluating geometry with textureless renders, its own implementation of Dream Fields performs nearly at chance, while DreamFusion is consistent with captions 58.5% of the time.

The paper says SDS is not a perfect loss function when applied to image sampling and often produces oversaturated and oversmoothed results relative to ancestral sampling, and that 2D image samples produced using SDS tend to lack diversity compared to ancestral sampling while its 3D results exhibit few differences across random seeds.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.