Text-to-Image: Generating Images from Prompts

Definition

Text-to-image generates an image from a text description. The text description is also known as a prompt. The most common text-to-image models are Stable Diffusion v1.5, Stable Diffusion XL (SDXL), and Kandinsky 2.2.

ControlNet models or adapters can be used with text-to-image models for more direct control in generating images. The results from each model are slightly different because of their architecture and training process, but no matter which model is chosen, usage is more or less the same.

At a glance

A diffusion model takes a prompt and some random initial noise, and iteratively removes the noise to construct an image. The denoising process is guided by the prompt, and once it ends after a predetermined number of time steps, the image representation is decoded into an image.

A checkpoint is loaded into the auto pipeline class, which automatically detects the appropriate pipeline class based on the checkpoint.
A prompt is passed to the pipeline to generate an image.
The height and width parameters control the height and width, in pixels, of the generated image.
The guidance scale parameter affects how much the prompt influences image generation.
A negative prompt steers the model away from things you do not want the model to generate.
A generator object enables reproducibility in a pipeline by setting a manual seed.
Diffusion models are large and iterative denoising is computationally expensive and intensive, but this does not mean powerful or many GPUs are needed to use them.

The source does not state the order of these points, so they are listed without arrows.

How it works

A number of parameters can be configured in the pipeline that affect how an image is generated. A lower guidance scale value gives the model creativity to generate images that are more loosely related to the prompt, while higher values push the model to follow the prompt more closely, and if the value is too high, some artifacts may be observed in the generated image. Just like a prompt guides generation, a negative prompt steers the model away from things you do not want the model to generate.

A negative prompt is commonly used to improve overall image quality by removing poor or bad image features such as low resolution or bad details. A negative prompt can also be used to remove or modify the content and style of an image. Creating an image with a generator should return the same result each time instead of randomly generating a new image.

Prompt weighting is a technique for increasing or decreasing the importance of concepts in a prompt to emphasize or minimize certain features in an image. ControlNet models are auxiliary models or adapters that are finetuned on top of text-to-image models, such as Stable Diffusion v1.5. With ControlNet, an additional conditioning input image is added to the model.

For example, if an image of a human pose, usually represented as multiple keypoints connected into a skeleton, is provided as a conditioning input, the model generates an image that follows the pose, and a Multi-ControlNet can combine multiple conditioning inputs.

Example

Stable Diffusion v1.5 is a latent diffusion model initialized from Stable Diffusion v1-4 and finetuned for 595K steps on 512x512 images from the LAION-Aesthetics V2 dataset. SDXL is a much larger version of the previous Stable Diffusion models and involves a two-stage model process that adds even more details to an image. The Kandinsky model is a bit different from the Stable Diffusion models because it also uses an image prior model to create embeddings that better align text and images in the diffusion model.

By default, the Stable Diffusion v1.5 model outputs 512x512 images, but this can be changed to any size that is a multiple of 8. SDXL's default image size is 1024x1024, and using lower height and width values may result in lower quality images. Other models may have different default image sizes depending on the image sizes in the training dataset.

Each ControlNet model is pretrained on a particular type of conditioning image to generate new images that resemble it. For example, a ControlNet model pretrained on depth maps, given a depth map as a conditioning input, generates an image that preserves the spatial information in it.

Related tools listed on this site.

Stable Diffusion

This Stable Diffusion review is limited to the v1 text-to-image research model described by the official CompVis repository. The README presents text-conditioned generation, image-to-image modification, reference sampling scripts and Diffusers integration.