中文

PowerPaint: A Versatile Image Inpainting Model

An image inpainting model supporting text-guided object inpainting, object removal, shape-guided object insertion, and outpainting

Verified

Core facts

Read by a script from the official API, not rewritten by a model.

License
MIT
Languages
Python
Maintainer
open-mmlab
Verified

What it is

The project says PowerPaint is a high-quality versatile image inpainting model that supports text-guided object inpainting, object removal, shape-guided object insertion, and outpainting at the same time. It achieves this by learning with tailored task prompts for different inpainting tasks. The team open sources the model weights and code for PowerPaint v2. The team retrained a new PowerPaint, taking inspiration from Brushnet.

Main features

  1. PowerPaint is compatible with ControlNet.

  2. Therefore, users can generate objects with a control image.

  3. PowerPaint supports shape-guided object inpainting, which lets users control how well the generated objects fit the shape of the masks.

  4. The team preserves the cross-attention layer that was deleted by BrushNet for the task prompts input.

Getting started

  1. Users can construct a conda environment from scratch by running a command.

  2. Users can launch the Gradio interface for PowerPaint by running a command.

  3. For the BrushNet-based PowerPaint, users can run a command.

  4. After launching the Gradio interface, users can insert objects into images by uploading an image, drawing the mask, selecting the text-guided object inpainting tab and inputting the text prompt.

  5. The model will then generate the output image.

  6. For image outpainting, select the image outpainting tab and adjust the sliders for horizontal expansion ratio and vertical expansion ratio, and PowerPaint will extend the image.

From the README · bash
# Clone the Repository
git clone git@github.com:open-mmlab/PowerPaint.git

# Create Virtual Environment with Conda
conda create --name ppt python=3.9
conda activate ppt

# Install Dependencies
pip install -r requirements/requirements.txt
From the README · bash
conda env create -f requirements/ppt.yaml
conda activate ppt
From the README · bash
# Set up Git LFS
conda install git-lfs
git lfs install

# Clone PowerPaint Model
git lfs clone https://huggingface.co/JunhaoZhuang/PowerPaint-v1/ ./checkpoints/ppt-v1

python app.py --share
From the README · bash
# Clone PowerPaint Model
git lfs clone https://huggingface.co/JunhaoZhuang/PowerPaint_v2/ ./checkpoints/ppt-v2

python app.py --share --version ppt-v2 --checkpoint_dir checkpoints/ppt-v2

Who it is for

For object removal, users do not need to input any prompts. If undesired objects appear in the masked area, this can be addressed by increasing the Guidance Scale. For image outpainting, users do not need to input any text prompt.

Users can adjust the fitting degree slider to control the shape of the generated object. The team recommends a fitting degree of 0.5-0.6 for generating objects that are not constrained by the mask shape. To generate objects that fit the mask shape, users can use a fitting degree of 0.8-0.95.

If a high fitting degree is used for a square rabbit, the result may look funny. The team refactored the code for better support of both the U-Net-based and Brush-Net network architectures. The team open-sourced the model weights for PowerPaint v2-1, rectifying some issues present during the training process of version 2.

The logical error in loading ControlNet has been rectified. The team announced the release of more stable model weights.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.