We’ll also be decoupling the tokenization process from the model as the name suggests. Through out this new model, we’ll replace all predict_on_batch() calls with graph compatible _call_() calls. In order to patch up the eager mode StableDiffusion model, we’ll create a new model called StableDiffusionNoTokenizer. The current version uses predict_on_batch in order to generate an image, which is not supported by TensorFlow graph building.The current version of the model uses a TensorFlow graph incompatible tokenizer, so it needs to be brought out of the packaged model and used in a separate step.One crucial requirement to do this is making sure all operations used are TensorFlow graph compatible. Our goal here is to save the StableDiffusionmodel in the SavedModel format the go-to standard for serializing TensorFlow models. You can open up a notebook and play with the model to familiarize yourself. You can find a tutorial here, which explains how to use the StableDiffusion in keras_cv. In order to construct a Stable Diffusion model, we’ll be using the keras_cv library, which includes a collection of popular deep learning vision models for image classification, segmentation, generative AI, etc.

Then a model is trained to predict the original image from noise.Īll of the above components work cohesively to achieve this idea. By reversing the order of the process, you can have an input (noise) and a target (original image). If you add a bit of noise to an image gradually for many steps, you will end up with an image containing noise. The principal ground-shattering idea behind stable diffusion (diffusion models) is,

Image encoder (used for the in-painting feature - we’ll be ignoring this for this exercise).

Decoder - Takes in the final latent image and produces an actual image.

Diffusion model - Takes in the text embedding and a latent image (initially noise) as an input and incrementally refine the latent image to encoder more and more useful information (visually pleasing).

Text encoder - Takes in the tokenized text and produces a text embedding.

Tokenizer - Tokenizes a given string to a list of tokens (numerical IDs).

There are five main components that builds up the Stable Diffusion model:

Some images generated by the deployed Stable Diffusion model Preparing the model artifacts What is Stable Diffusion anyway?

Terraform up and running