Virtual Try-On: the IDM–VTON paper

The paper considers image-based virtual try-on, which renders an image of a person wearing a curated garment, given images of the person and the garment.

Verified

Definition

The paper describes image-based virtual try-on (VTON) as a computer vision task whose goal is to render a visual of an arbitrary person dressing in a specific garment given by images. The paper considers image-based virtual try-on, which renders an image of a person wearing a curated garment, given a pair of images depicting the person and the garment, respectively. The paper states that previous works adapt existing exemplar-based inpainting diffusion models for virtual try-on to improve the naturalness of the generated visuals compared to other methods (e.g., GAN-based), but they fail to preserve the identity of the garments; recent diffusion-based VTON methods utilize the generative prior of pretrained text-to-image diffusion models to enhance the naturalness of try-on images.

The paper proposes a novel diffusion model that improves garment fidelity and generates authentic virtual try-on images. The paper names its method IDM–VTON, which uses two different modules to encode the semantics of the garment image: given the base UNet of the diffusion model, high-level semantics extracted from a visual encoder are fused to the cross-attention layer, and low-level features extracted from a parallel UNet are fused to the self-attention layer.

At a glance

The paper's model includes the base UNet (TryonNet), the image prompt adapter (IP-Adapter) and the garment UNet feature encoder (GarmentNet). In its implementation, the paper uses the SDXL inpainting model for TryonNet, a pretrained IP-Adapter for the image adapter and the UNet of SDXL for GarmentNet.

  • TryonNet is the base UNet that processes the masked person image with its pose information.
  • IP-Adapter extracts the high-level semantics of the garment.
  • GarmentNet extracts the low-level features of the garment.
  • The features from GarmentNet are fused within the self-attention layer of TryonNet and then processed with the features of IP-Adapter through the cross-attention layer.

The source does not state the order of these points, so they are listed without arrows.

How it works

The paper designs attention modules to condition garment images, consisting of an image prompt adapter that encodes high-level semantics of the garment and a UNet encoder called GarmentNet that extracts low-level features to preserve fine-grained details. The paper states that the CLIP image encoder lacks extracting the low-level features of a garment. To encode the garment image, the paper uses a frozen CLIP image encoder to extract features and fine-tunes feature projection layers and cross-attention layers, which are initialized with pretrained IP-Adapters.

The paper provides a comprehensive caption that describes the detail of a garment, such as shapes or textures. The paper also provides detailed textual prompts for both garment and person images to enhance the authenticity of the generated visuals. The paper shows that providing detailed captions for the garment image helps retain the prior knowledge of T2I diffusion models.

The paper proposes customization of IDM–VTON by fine-tuning TryonNet with a pair of garment and person images, where the person in the person image is already wearing the garment in the garment image. If only the person image is available, the paper segments the garment from it and whites out the background to obtain the garment image. Under this pairing condition, the paper also proposes to customize its model using a single pair of garment and person images, which it says further improves visual quality particularly in the wild scenario, and presents this customization method using a pair of person-garment images as significantly improving fidelity and authenticity.

For training, the paper uses the Adam optimizer with a fixed learning rate of 1e-5 over 130 epochs with batch size of 24, and during inference it generates images using the DDPM scheduler with 30 steps. For customization, the paper fine-tunes its model using the Adam optimizer with a fixed learning rate of 1e-6 for 100 steps.

Example

The paper collected the In-the-Wild dataset, which contains garments with intricate patterns and human images with various poses and gestures, largely different from the training data. The In-the-Wild dataset comprises multiple human images wearing each target garment. This dataset consists of 62 images of upper garments and 312 images of people wearing those garments, with 4-6 images of a person wearing the same garment.

The paper compares its method with the GAN-based methods HR-VITON and GP-VTON. For quantitative evaluation, the paper measures low-level reconstruction scores such as LPIPS and SSIM and high-level image similarity using the CLIP image similarity score, and it computes the FID score on the VITON-HD and DressCode test datasets to evaluate authenticity. By customizing IDM–VTON, the paper obtains images with a high degree of consistency to the garment but finds the customization does not work well for StableVITON, and it says the proposed customization method demonstrates effectiveness in a real-world scenario.

In the customization ablation, the paper finds that fine-tuning with LoRA is more efficient but underperforms fine-tuning decoder layers, and it states that IDM–VTON struggles to preserve human attributes on masked regions such as tattoos or skin moles.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.