Inpainting: replacing or editing specific areas of an image

Definition

Inpainting replaces or edits specific areas of an image. Inpainting relies on a mask to determine which regions of an image to fill in; the area to inpaint is represented by white pixels and the area to keep is represented by black pixels. The white pixels are filled in by the prompt.

You can inpaint on your own images, but you need to create a mask image for it. In the official Diffusers inpainting guide's example Space, you first upload a base image to inpaint on and then use the Space's sketch tool to draw a mask.

At a glance

Image-to-image generates a new image that resembles the existing provided image.

Load the base and mask images.
Create a prompt to inpaint the image with and pass it to the inpainting pipeline together with the base and mask images.
Chaining a text-to-image and inpainting pipeline allows you to inpaint the generated image, and you do not have to provide a base image to begin with.
If Stable Diffusion v1.5 is used for both pipelines, everything can be kept in latent space because both use AutoencoderKL.
An inpainting pipeline can be chained before another pipeline like image-to-image or an upscaler to improve the quality.
Prompt weighting provides a quantifiable way to scale the representation of concepts in a prompt.
A ControlNet accepts an additional conditioning image input that guides the diffusion model to preserve the features in it.
Strength and guidance scale can be used together for more control over how expressive the model is.

The source does not state the order of these points, so they are listed without arrows.

How it works

The blur method provides an option for how to blend the original image and inpaint area. Increasing the blur factor increases the amount of blur applied to the mask edges, softening the transition between the original image and inpaint area. A low or zero blur factor preserves the sharper edges of the mask.

Strength is a measure of how much noise is added to the base image, which influences how similar the output is to the base image. Guidance scale affects how aligned the text prompt and generated image are. A high guidance scale value means the prompt and generated image are closely aligned, so the output is a stricter interpretation of the prompt, while a low value means they are more loosely aligned, so the output may be more varied from the prompt.

A negative prompt assumes the opposite role of a prompt; it guides the model away from generating certain things in an image.

Example

Stable Diffusion Inpainting is a latent diffusion model finetuned on 512x512 images on inpainting. SDXL is a larger and more powerful version of Stable Diffusion v1.5. SDXL can follow a two-stage model process, though each model can also be used alone; the base model generates an image, and a refiner model takes that image and further enhances its details and quality.

The Kandinsky model family is similar to SDXL because it uses two models as well; the image prior model creates image embeddings, and the diffusion model generates images from them. The trade-off of using a non-inpaint specific checkpoint is that the overall image quality may be lower, but it generally tends to preserve the mask area, which is why the mask outline can be seen. The inpaint specific checkpoints are intentionally trained to generate higher quality inpainted images, and that includes creating a more natural transition between the masked and unmasked areas.

As a result, these checkpoints are more likely to change the unmasked area. If preserving the unmasked area is important for a task, the VaeImageProcessor.apply_overlay() method of Diffusers can force the unmasked area of an image to remain the same, at the expense of some more unnatural transitions between the masked and unmasked areas.

Related tools listed on this site.