![\"Image\"](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2023\/02\/Tom-Darmon.jpeg\")
![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2021\/04\/Medium-Blog.png\" "\\"Medium")
<\/span><\/div>Introduction<\/h2><\/div>Stable Diffusion is an image generation model. It was open sourced in 2022 and has gained popularity due to its ability to generate high quality images from text descriptions. Like other image generation models, such as Dall-E, Stable Diffusion uses machine learning techniques to generate images based on a given input.<\/p>\n
Mo\u00ebt Hennessy, the wine and spirits division of luxury conglomerate LVMH, manages a portfolio of more than 26 iconic brands such as Mo\u00ebt & Chandon, Hennessy, and Veuve Clicquot. Mo\u00ebt Hennessy has collaborated with Artefact<\/a> to investigate the potential uses of cutting-edge technology in marketing content generation. With a focus on privacy and security, the team decided to explore the deployment of Stable Diffusion on Google Cloud Platform (GCP) to allow Mo\u00ebt Hennessy to fine-tune and run the model within their own infrastructure, providing a seamless experience from model fine-tuning to API exposure.<\/p>\nBefore you begin, it is important to note that this article assumes that you have prior knowledge of Google Cloud Platform (GCP) and specifically Vertex AI. This includes concepts such as model registry and Vertex endpoints. Additionally, you need to have prior experience with Docker to follow some of the steps. If you are not familiar with these concepts, it is recommended that you familiarize yourself with them before proceeding.
\nAdditionally, in order to download Stable Diffusion weights you need to have a huggingface account, if you don\u2019t have one already you can create one easily on\u00a0huggingface\u2019s website<\/a>.
\nWith that being said, let\u2019s begin !<\/p>\n<\/div>Setup and weights download<\/h2><\/div>I recommend cloning the github repository<\/a>\u00a0I prepared in order to follow the steps of the article.<\/p>\nIt is always important to create a virtual environment to install the packages. Personally, I will use anaconda and install the requirements.txt with every dependency:<\/p>\n<\/div>
conda create -n stable_diffusion \u2014 no-default-packages python=3.8 -y
\nconda activate stable_diffusion
\npip install -r src\/requirements.txt<\/div>\n<\/div>You are now ready to download the weights of Stable Diffusion, in this article we will use\u00a0Stable Diffusion 1.5<\/a>. You need to accept the license on the model page, else you will have errors when downloading the model weights.
\nGo to your account \u2192 settings \u2192 access token \u2192 new token (read access)<\/p>\n<\/div>![\"Go](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2023\/02\/image-vertex1.png\" "\\"image-vertex1\\"")
<\/span><\/div>You can add the token as an environment variable. Personally, I recommend using a .env file and loading the environment variable using the\u00a0python-dotenv library<\/a>.
\nYou have to navigate to `src\/stable_diffusion` and run:<\/p>\n<\/div>python download_model.py<\/div>\n<\/div>This will download the weights inside<\/p>\n
`src\/stable_diffusion\/external_files\/model_weights`.<\/div>\n<\/div>Torchserve framework<\/h2><\/div>Torchserve<\/a>\u00a0is a framework for serving PyTorch models. It allows you to deploy PyTorch models in a production environment and provides features such as model versioning and multi-model serving. It is designed to be easy to use and allows you to focus on building and deploying your models, rather than worrying about infrastructure.<\/p>\n<\/div>1. Creating a handler respecting TorchServe format<\/h3><\/div>A custom handler is a Python class that defines how to pre-process input data, how to run the model, and how to post-process the output. To create a custom handler for your model, you will need to create a Python class that follows the TorchServe format.
\nA custom handler for Stable Diffusion is already given inside the TorchServe repository. But Vertex endpoint expects a specific format for the requests, so we need to adapt the preprocess() method of the handler to account for Vertex format. You can use the modified version of the handler named `stable_diffusion_handler.py<\/a>` given inside the github repository of this article.<\/p>\n<\/div>2. Creating the .mar file<\/h3><\/div>Once you have created your custom handler, you will need to package it along with any dependencies and the model itself into a .mar file using the model-archiver tool. The model-archiver tool is a command-line tool that allows you to package your model, handler, and dependencies into a single file.<\/p>\n
This will create a .mar file called output.mar that contains your model, handler, and dependencies.<\/p>\n
You might need to edit the path based on where the files are on your machine:<\/p>\n<\/div>
torch-model-archiver
\n–model-name stable-diffusion
\n–version 1.0
\n–handler stable_diffusion\/stable_diffusion_handler.py
\n–export-path stable_diffusion\/model-store
\n–extra-files stable_diffusion\/external_files <\/div>\n<\/div>3. Running your TorchServe server<\/h3><\/div>Once you have created your .mar file, you can start the TorchServe server using the torchserve command. To do this, you will need to run the following command:<\/p>\n<\/div>
torchserve
\n\u2014 start
\n\u2014 ts-config=config.properties
\n\u2014 models=stable-diffusion.mar
\n\u2014 model-store=stable_diffusion\/model-store<\/div>\n<\/div>The config.properties let you specify the configurations of your TorchServe server, such as port for inference, health checks, number of workers, etc..
\nWARNING: All the scripts need to be run where the file is located to avoid any path errors.<\/em><\/p>\n<\/div>Run the TorchServe server locally<\/h2><\/div>It is important to test the code locally before starting to dockerize our deployment, I already prepared a bash script that will create the .mar file and start the TorchServe server.<\/p>\n
You can run it with :<\/p>\n
<\/pre>\n<\/div>bash serve_locally.sh<\/div>\n<\/div>You need to wait a few minutes for the server to initialize and the worker to load the model, you can start to run inference if you see the following log:<\/p>\n<\/div>
2023\u201301\u201305T15:34:52,842 [DEBUG] W-9000-stable-diffusion_1.0
\norg.pytorch.serve.wlm.WorkerThread\u2014 W-9000-stable-diffusion_1.0
\nState change WORKER_STARTED -> WORKER_MODEL_LOADED<\/div>\n<\/div>You can then use the following code to make inference requests to your model:<\/p>\n<\/div>
import requests
\nprompt = “a photo of an astronaut riding a horse on mars”
\nURL = “http:\/\/localhost:7080\/predictions\/stable-diffusion”
\nresponse = requests.post(URL, data=prompt)<\/div>\n<\/div>You can check that the server received your request by looking at the server logs:<\/p>\n<\/div>
2023\u201301\u201305T15:35:43,765 [INFO ] W-9000-stable-diffusion_1.0-stdout
\nMODEL_LOG \u2014 Backend received inference at: 1672929343<\/div>\n<\/div>Stable Diffusion requires a GPU to run smoothly, so you won\u2019t have any output for the moment. You can stop the torchserve server with `torhcserve \u2013stop`.<\/p>\n<\/div>
Dockerize torchserve<\/h2><\/div>Your TorchServe server is working locally, to deploy Stable Diffusion on Vertex AI, you will need to dockerize it. This means creating a Docker image that contains the model, the custom handler, and all the necessary dependencies. This is simply all the steps we have done above inside a Dockerfile.
\nLuckily, it is already prepared and ready to use here :\u00a0Dockerfile<\/a>.<\/p>\nIt\u2019s important to run the container locally to check that it is working properly. I\u2019m going to build it locally, but you can build it with cloud build and pull it on your machine.<\/p>\n
Build the image locally (you need Docker daemon running):<\/p>\n<\/div>
build -t serve_sd .<\/div>\n<\/div>The image build will take 20 to 30 minutes, the build phase is long because the weights of the model need to be copied inside the image before being packaged by the model-archiver.<\/p>\n
You can run a docker container and listen on port 7080 with:<\/p>\n<\/div>
docker run -p 7080:7080 serve_sd<\/div>\n<\/div>To check that everything is working properly, you can wait until the worker has loaded the model and run the same inference code as before since we are using the same 7080 port.<\/p>\n<\/div>
Deployment on Vertex AI<\/h2><\/div>Now that the Dockerfile is ready and working, we need to:<\/p>\n<\/div>
- <\/i><\/span>Build the image using Cloud Build inside google container registry (gcr)<\/div><\/li>
- <\/i><\/span>\n
Upload the image of our custom model inside Vertex AI model registry<\/p>\n<\/div><\/li>
- <\/i><\/span>Create a Vertex AI endpoint<\/div><\/li>
- <\/i><\/span>\n
Attach the model to the endpoint<\/p>\n<\/div><\/li><\/ul>
This is exactly what the\u00a0deploy.sh<\/a>\u00a0bash script is going to do, you can run it with:<\/p>\n<\/div>bash deploy.sh<\/div>\n<\/div>The deployment takes around 1 hour with good internet speed:<\/p>\n<\/div>
- <\/i><\/span>Sending the 8 GB of model weights to Cloud Build can take from a few minutes to hours depending on your internet speed<\/div><\/li>
- <\/i><\/span>\n
Building the image takes around 20 minutes<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
Uploading the model takes around 5 minutes<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
Creating the endpoint takes around 5 minutes<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
Attaching the model to the endpoint takes from 30 to 40 minutes<\/p>\n<\/div><\/li><\/ul>
After the model is successfully attached to the endpoint, you can query the endpoint with the following code:<\/p>\n<\/div>
from google.cloud import aiplatform as aip<\/div>\nPROJECT_NAME = \"\"
\nREGION = \"\"
\nENDPOINT_ID = \"\"
\naip.init(project=PROJECT_NAME, location=REGION)
\nendpoint = aip.Endpoint(endpoint_name=ENDPOINT_ID)
\ntext_input = \"\"\"A bottle of aged and exclusive cognac
\nstands on a reflective surface, in front of a vibrant bar,
\nhyper detailed, 4K, bokeh\"\"\"<\/p>\n
def query_endpoint(endpoint, text_input):
\npayload =
\nresponse = endpoint.predict(instances=[payload])
\nreturn response<\/p>\n
image = query_endpoint(endpoint, text_input)<\/div>\n<\/div>![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2023\/02\/1_uS37ust4GEzk7-0OBYNReg.webp\" "\\"1_uS37ust4GEzk7-0OBYNReg\\"")
<\/span><\/div>A bottle of aged and exclusive cognac stands on a reflective surface, in front of a vibrant bar, hyper detailed, 4K, bokeh \u2014 Stable Diffusion 1.5<\/em><\/p>\n<\/div>The inference should take between 10 and 15 seconds on T4 GPU. You can improve the speed by choosing a better GPU, you can change the ACCELERATOR_TYPE variable inside deploy.sh<\/a>.<\/p>\n<\/div>Optional: If the image size is too big for the endpoint, or you want to store the image history.<\/h2><\/div>You might want to keep track of the history of the images, or you are having errors due to the size limit of 1.5 MB for the endpoint response. In this case, I recommend using the post-process method of the handler to save the image inside GCS and return only the GCS path of the image.<\/p>\n
Stable Diffusion is an image generation model. It was open sourced in 2022 and has gained popularity due to its ability to generate high quality images from text descriptions. Like other image generation models, such as Dall-E, Stable Diffusion uses machine learning techniques to generate images based on a given input.<\/p>\n
Mo\u00ebt Hennessy, the wine and spirits division of luxury conglomerate LVMH, manages a portfolio of more than 26 iconic brands such as Mo\u00ebt & Chandon, Hennessy, and Veuve Clicquot. Mo\u00ebt Hennessy has collaborated with Artefact<\/a> to investigate the potential uses of cutting-edge technology in marketing content generation. With a focus on privacy and security, the team decided to explore the deployment of Stable Diffusion on Google Cloud Platform (GCP) to allow Mo\u00ebt Hennessy to fine-tune and run the model within their own infrastructure, providing a seamless experience from model fine-tuning to API exposure.<\/p>\n Before you begin, it is important to note that this article assumes that you have prior knowledge of Google Cloud Platform (GCP) and specifically Vertex AI. This includes concepts such as model registry and Vertex endpoints. Additionally, you need to have prior experience with Docker to follow some of the steps. If you are not familiar with these concepts, it is recommended that you familiarize yourself with them before proceeding. I recommend cloning the github repository<\/a>\u00a0I prepared in order to follow the steps of the article.<\/p>\n It is always important to create a virtual environment to install the packages. Personally, I will use anaconda and install the requirements.txt with every dependency:<\/p>\n<\/div> You are now ready to download the weights of Stable Diffusion, in this article we will use\u00a0Stable Diffusion 1.5<\/a>. You need to accept the license on the model page, else you will have errors when downloading the model weights. You can add the token as an environment variable. Personally, I recommend using a .env file and loading the environment variable using the\u00a0python-dotenv library<\/a>. This will download the weights inside<\/p>\n Torchserve<\/a>\u00a0is a framework for serving PyTorch models. It allows you to deploy PyTorch models in a production environment and provides features such as model versioning and multi-model serving. It is designed to be easy to use and allows you to focus on building and deploying your models, rather than worrying about infrastructure.<\/p>\n<\/div> A custom handler is a Python class that defines how to pre-process input data, how to run the model, and how to post-process the output. To create a custom handler for your model, you will need to create a Python class that follows the TorchServe format. Once you have created your custom handler, you will need to package it along with any dependencies and the model itself into a .mar file using the model-archiver tool. The model-archiver tool is a command-line tool that allows you to package your model, handler, and dependencies into a single file.<\/p>\n This will create a .mar file called output.mar that contains your model, handler, and dependencies.<\/p>\n You might need to edit the path based on where the files are on your machine:<\/p>\n<\/div> Once you have created your .mar file, you can start the TorchServe server using the torchserve command. To do this, you will need to run the following command:<\/p>\n<\/div> The config.properties let you specify the configurations of your TorchServe server, such as port for inference, health checks, number of workers, etc.. It is important to test the code locally before starting to dockerize our deployment, I already prepared a bash script that will create the .mar file and start the TorchServe server.<\/p>\n You can run it with :<\/p>\n You need to wait a few minutes for the server to initialize and the worker to load the model, you can start to run inference if you see the following log:<\/p>\n<\/div> You can then use the following code to make inference requests to your model:<\/p>\n<\/div> You can check that the server received your request by looking at the server logs:<\/p>\n<\/div> Stable Diffusion requires a GPU to run smoothly, so you won\u2019t have any output for the moment. You can stop the torchserve server with `torhcserve \u2013stop`.<\/p>\n<\/div> Your TorchServe server is working locally, to deploy Stable Diffusion on Vertex AI, you will need to dockerize it. This means creating a Docker image that contains the model, the custom handler, and all the necessary dependencies. This is simply all the steps we have done above inside a Dockerfile. It\u2019s important to run the container locally to check that it is working properly. I\u2019m going to build it locally, but you can build it with cloud build and pull it on your machine.<\/p>\n Build the image locally (you need Docker daemon running):<\/p>\n<\/div> The image build will take 20 to 30 minutes, the build phase is long because the weights of the model need to be copied inside the image before being packaged by the model-archiver.<\/p>\n You can run a docker container and listen on port 7080 with:<\/p>\n<\/div> To check that everything is working properly, you can wait until the worker has loaded the model and run the same inference code as before since we are using the same 7080 port.<\/p>\n<\/div> Now that the Dockerfile is ready and working, we need to:<\/p>\n<\/div> Upload the image of our custom model inside Vertex AI model registry<\/p>\n<\/div><\/li> Attach the model to the endpoint<\/p>\n<\/div><\/li><\/ul> This is exactly what the\u00a0deploy.sh<\/a>\u00a0bash script is going to do, you can run it with:<\/p>\n<\/div> The deployment takes around 1 hour with good internet speed:<\/p>\n<\/div> Building the image takes around 20 minutes<\/p>\n<\/div><\/li> Uploading the model takes around 5 minutes<\/p>\n<\/div><\/li> Creating the endpoint takes around 5 minutes<\/p>\n<\/div><\/li> Attaching the model to the endpoint takes from 30 to 40 minutes<\/p>\n<\/div><\/li><\/ul> After the model is successfully attached to the endpoint, you can query the endpoint with the following code:<\/p>\n<\/div> PROJECT_NAME = \"\" def query_endpoint(endpoint, text_input): A bottle of aged and exclusive cognac stands on a reflective surface, in front of a vibrant bar, hyper detailed, 4K, bokeh \u2014 Stable Diffusion 1.5<\/em><\/p>\n<\/div> The inference should take between 10 and 15 seconds on T4 GPU. You can improve the speed by choosing a better GPU, you can change the ACCELERATOR_TYPE variable inside deploy.sh<\/a>.<\/p>\n<\/div> You might want to keep track of the history of the images, or you are having errors due to the size limit of 1.5 MB for the endpoint response. In this case, I recommend using the post-process method of the handler to save the image inside GCS and return only the GCS path of the image.<\/p>\n
\nAdditionally, in order to download Stable Diffusion weights you need to have a huggingface account, if you don\u2019t have one already you can create one easily on\u00a0huggingface\u2019s website<\/a>.
\nWith that being said, let\u2019s begin !<\/p>\n<\/div>Setup and weights download<\/h2><\/div>
\nconda activate stable_diffusion
\npip install -r src\/requirements.txt<\/div>\n<\/div>
\nGo to your account \u2192 settings \u2192 access token \u2192 new token (read access)<\/p>\n<\/div><\/span><\/div>
\nYou have to navigate to `src\/stable_diffusion` and run:<\/p>\n<\/div>Torchserve framework<\/h2><\/div>
1. Creating a handler respecting TorchServe format<\/h3><\/div>
\nA custom handler for Stable Diffusion is already given inside the TorchServe repository. But Vertex endpoint expects a specific format for the requests, so we need to adapt the preprocess() method of the handler to account for Vertex format. You can use the modified version of the handler named `stable_diffusion_handler.py<\/a>` given inside the github repository of this article.<\/p>\n<\/div>2. Creating the .mar file<\/h3><\/div>
\n–model-name stable-diffusion
\n–version 1.0
\n–handler stable_diffusion\/stable_diffusion_handler.py
\n–export-path stable_diffusion\/model-store
\n–extra-files stable_diffusion\/external_files <\/div>\n<\/div>3. Running your TorchServe server<\/h3><\/div>
\n\u2014 start
\n\u2014 ts-config=config.properties
\n\u2014 models=stable-diffusion.mar
\n\u2014 model-store=stable_diffusion\/model-store<\/div>\n<\/div>
\nWARNING: All the scripts need to be run where the file is located to avoid any path errors.<\/em><\/p>\n<\/div>Run the TorchServe server locally<\/h2><\/div>
<\/pre>\n<\/div>
\norg.pytorch.serve.wlm.WorkerThread\u2014 W-9000-stable-diffusion_1.0
\nState change WORKER_STARTED -> WORKER_MODEL_LOADED<\/div>\n<\/div>
\nprompt = “a photo of an astronaut riding a horse on mars”
\nURL = “http:\/\/localhost:7080\/predictions\/stable-diffusion”
\nresponse = requests.post(URL, data=prompt)<\/div>\n<\/div>
\nMODEL_LOG \u2014 Backend received inference at: 1672929343<\/div>\n<\/div>Dockerize torchserve<\/h2><\/div>
\nLuckily, it is already prepared and ready to use here :\u00a0Dockerfile<\/a>.<\/p>\nDeployment on Vertex AI<\/h2><\/div>
\nREGION = \"\"
\nENDPOINT_ID = \"\"
\naip.init(project=PROJECT_NAME, location=REGION)
\nendpoint = aip.Endpoint(endpoint_name=ENDPOINT_ID)
\ntext_input = \"\"\"A bottle of aged and exclusive cognac
\nstands on a reflective surface, in front of a vibrant bar,
\nhyper detailed, 4K, bokeh\"\"\"<\/p>\n
\npayload =
\nresponse = endpoint.predict(instances=[payload])
\nreturn response<\/p>\n<\/span><\/div>Optional: If the image size is too big for the endpoint, or you want to store the image history.<\/h2><\/div>
![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2021\/03\/medium.png\" "\\"medium\\"")
<\/span><\/div>