Image-to-Image: Generating Images from an Initial Image

Definition

Image-to-image is similar to text-to-image, but in addition to a prompt, you can also pass an initial image as a starting point for the diffusion process. Stable Diffusion v1.5 is a latent diffusion model initialized from an earlier checkpoint, and further finetuned for 595K steps on 512x512 images. The Kandinsky model is different from the Stable Diffusion models because it uses an image prior model to create image embeddings.

The Diffusers image-to-image documentation says the Kandinsky image prior's embeddings help create a better alignment between text and images, allowing the latent diffusion model to generate better images. Chaining a text-to-image and an image-to-image pipeline allows you to generate an image from text and use the generated image as the initial image for the image-to-image pipeline.

At a glance

Prompt weighting allows you to scale the representation of each concept in a prompt.

Load a checkpoint into the AutoPipelineForImage2Image class; this pipeline automatically handles loading the correct pipeline class based on the checkpoint.
Load an image to pass to the pipeline.
Pass a prompt and an image to the pipeline to generate an image.
The initial image is encoded to latent space and noise is added to it.
Then the latent diffusion model takes a prompt and the noisy latent image, predicts the added noise, and removes the predicted noise from the initial latent image to get the new latent image.
Lastly, a decoder decodes the new latent image back into an image.
The Compel library provides a simple syntax for adjusting prompt weights and generating the embeddings.
AutoPipelineForImage2Image has a prompt_embeds parameter (and a negative_prompt_embeds parameter when a negative prompt is used) where the embeddings can be passed to replace the prompt parameter.

The source does not state the order of these points, so they are listed without arrows.

How it works

A higher strength value gives the model more creativity to generate an image that is different from the initial image, and a strength value of 1.0 means the initial image is more or less ignored. A lower strength value means the generated image is more similar to the initial image. The strength and num_inference_steps parameters are related because strength determines the number of noise steps to add.

For example, if num_inference_steps is 50 and strength is 0.8, this means adding 40 (50 * 0.8) steps of noise to the initial image and then denoising for 40 steps to get the newly generated image. The guidance_scale parameter is used to control how closely aligned the generated image and the text prompt are. A higher guidance_scale value means the generated image is more aligned with the prompt, while a lower value means the generated image has more space to deviate from the prompt.

The guidance_scale parameter can be combined with strength for more precise control over how expressive the model is. A negative prompt conditions the model to not include things in an image, and it can be used to improve image quality or modify an image. When chained pipelines keep their outputs in latent space to avoid an unnecessary decode-encode step, this only works if the chained pipelines use the same VAE.

Example

Another way to chain an image-to-image pipeline is with an upscaler and a super-resolution pipeline to increase the level of details in an image. Chain it to an upscaler pipeline to increase the image resolution. Finally, chain it to a super-resolution pipeline to further enhance the resolution. A new style can be applied to the image generated from the ControlNet by chaining it with an image-to-image pipeline.

Related tools listed on this site.

Stable Diffusion

This Stable Diffusion review is limited to the v1 text-to-image research model described by the official CompVis repository. The README presents text-conditioned generation, image-to-image modification, reference sampling scripts and Diffusers integration.