![\"Image\"](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2021\/04\/amale-elhamri.jpeg\")
![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2021\/04\/Medium-Blog.png\" "\\"Medium")
<\/span><\/div>TL;DR<\/h2><\/div>In this second article of the series of two, I will dive into the deployment and the serving of our models at scale. If you missed the first one about training a fastai model at scale on AI Platform Training, here is the link<\/a>.<\/p>\n<\/div>Serving a deep learning model can reveal several challenges among which:<\/p>\n<\/div>
- <\/i><\/span>\n
scaling resources on instances with or without accelerators (NVIDIA GPUs)<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
cost efficiency<\/p>\n<\/div><\/li><\/ul>
In this article, I will explain how I served a deep learning text classifier trained with the FastAI library following 2 main steps:<\/p>\n<\/div>
- <\/i><\/span>\n
Deploy fastai model using TorchServe<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
Host serving on GCP AI Platform Prediction<\/p>\n<\/div><\/li><\/ul>
All materials can be found in the g<\/a>ithub repository<\/a>. This repository was inspired by another project that aimed to deploy a fastai image classifier on AWS SageMaker Inference Endpoint here<\/a>.<\/p>\n<\/div>1- Deploy fastai model using TorchServe<\/h2><\/div>TorchServe makes it easy to deploy PyTorch models at scale in production environments. It removes the heavy lifting of developing your own client server architecture. The FastAI library is based on the PyTorch framework. It makes it possible to use this technology to serve fastai models by loading your fastai model as a pure pytorch object (remove fastai abstraction).<\/p>\n<\/div>
1\u20131 Export Model Weights from FastAI<\/h3><\/div>To do that, you need to restore the FastAI learner from the export pickle from the last post, and save its model weights with PyTorch.<\/p>\n<\/div>
import torch
\nfrom fastai.text import load_learner
\nfrom fastai.text.learner import get_c, get_text_vocab
\nlearn = load_learner(“fastai_cls.pkl”)
\nvocab_sz = len(_get_text_vocab(dls)) #dls is the dataloader you used for training
\nn_class = get_c(dls)
\nconfig = awd_lstm_clas_config.copy()
\ntorch.save(learn.model.state_dict(), “fastai_cls_weights.pth”)<\/div>\n<\/div>1\u20132 PyTorch Model from FastAI<\/h3><\/div>Once you\u2019ve exported your pytorch weights, you need to rebuild the model structure to be able to load your weights into it. You might have to dig a little bit in fastai source code to find your implementation but luckily, In Jupyter notebook, one can investigate the source code by adding ?? in front of a function name.<\/p>\n<\/div>
For text classifier, you can load a pure pytorch object by using the fastai get_text_classifier function<\/p>\n<\/div>
from fastai.text.learner import get_text_classifier
\nfrom fastai.text.all import AWD_LSTM
\ntorch_pure_model = get_text_classifier(AWD_LSTM, vocab_sz, n_class, config=config)<\/div>\n<\/div>1\u20133 Reproduce fastai preprocessing steps<\/h3><\/div>Once you have obtained your pytorch pure model, you need to apply the same preprocessing that was used for training. FastAI has a very handy method .predict that can be applied to a text (simple string object), that naturally reproduces training preprocessing and therefore removes risk of training serving skew.<\/p>\n<\/div>
text = “This was a very good movie”
\npred_fastai = learn.predict(text)
\npred_fastai
\n>>(Category tensor(1), tensor(1), tensor([0.0036, 0.9964]))<\/div>\n<\/div>In our case, we have to take this responsibility ourselves, since we need to get rid of fastai abstraction and work directly with PyTorch objects.<\/p>\n<\/div>
In my example, I used a spacy tokenizer so I reproduced fastai preprocessing as shown below:
\nimport torch<\/p>\n<\/div>
import torch
\nfrom fastai.text.core import Tokenizer, SpacyTokenizerfrom fastai.text.data import Numericalize
\nexample = “Hello, this is a test.”
\ntokenizer = Tokenizer(
\ntok=SpacyTokenizer(“en”)
\n)
\nnumericalizer = Numericalize(vocab=vocab)
\nexample_processed = numericalizer(tokenizer(example))
\nexample_processed
\n>>> tensor([ 4, 7, 26, 29, 16, 72, 69, 31])
\ninputs = example_processed.resize(1, len(example_processed))
\noutputs = model_torch.forward(inputs)[0]\npreds = torch.softmax(outputs, dim=-1) #You can use any activation function you need
\npreds
\n>>> tensor([[0.0036, 0.9964]], grad_fn=)<\/div>\n<\/div>As you can notice, the results I get using torch functions and learn.predict are the same because I managed to preserve the same preprocessing steps.<\/p>\n<\/div>
1\u20134 Deploy your model via torchserve<\/h3><\/div>In this section we deploy the PyTorch model to TorchServe. For installation, please refer to TorchServe Github Repository.
\nOverall, there are mainly 3 steps to use TorchServe:<\/p>\n
\n- Archive the model into *.mar.<\/li>\n
- Start the torchserve.<\/li>\n<\/ol>\n
Call the API and get the response.
\nIn order to archive the model, at least 2 files are needed in our case:<\/p>\n
\n- PyTorch model weights fastai_cls_weights.pth.<\/li>\n
- TorchServe custom handler.<\/li>\n<\/ol>\n<\/div>
Custom Handler<\/h3><\/div>As shown in \/deployment\/handler.py, the TorchServe handler accepts data and context. In our example, we define another helper Python class with 4 instance methods to implement: initialize, preprocess, inference and postprocess.<\/p>\n<\/div>
Now it\u2019s ready to setup and launch TorchServe.<\/p>\n<\/div>
TorchServe in Action<\/h3><\/div>Step 1: Archive the model PyTorch<\/h6><\/div>torch-model-archiver
\n\u2014-model-name=fastai_model
\n–version=1.0
\n–serialized-file=\/home\/model-server\/fastai_cls_weights.pth
\n\u2014- extra-files=\/home\/model-server\/config.py,\/home\/model-server\/vocab.json
\n–handler=\/home\/model-server\/handler.py
\n\u2014-export-path=\/home\/model-server\/model-store\/<\/div>\n<\/div>Step 2: Serve the Model<\/h6><\/div>torchserve –start –ncs –model-store model_store –models fastai_model.mar<\/div>\n<\/div>Step 3: Call API and Get the Response (here we use curl).<\/h6><\/div>curl -X POST -H \"Content-Type: application\/json\" -d '[\"this was a bad movie\"]' http:\/\/127.0.0.1:8080\/predictions\/fastai_model\n<\/div>\n<\/div>The first call would have longer latency due to model weights loading defined in initialize, but this will be mitigated from the second call onward.<\/p>\n<\/div>
2- Deployment to AI Platform Prediction<\/h2><\/div>In this section we deploy the FastAI trained model with TorchServe in GCP AI Platform Prediction using a customized Docker image. For more details about GCP AI Platform Prediction routines using custom containers please refer to this article. Note that this option is only available if you use AI Platform Prediction with regional endpoints.<\/p>\n<\/div>
Steps to deploy a fastai model on AI Platform Prediction:<\/p>\n<\/div>
First, create an AI Platform Prediction model on a regional endpoint:<\/p>\n<\/div>
gcloud beta ai-platform models create MODEL_NAME #eg: fastai_text_clf
\n–region=REGION #eg: europe-west1
\n–enable-logging
\n–enable-console-logging<\/div>\n<\/div>2\u20131 Build your docker image that will be used by your version<\/h3><\/div>- <\/i><\/span>\n
Create a folder model\/ in the root of the repository<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
Place your fastai model weights in model\/text\/ and name it fastai_cls_weights.pth<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
Create an artifact repository<\/p>\n<\/div><\/li><\/ul>
gcloud beta artifacts repositories create ARTIFACT_REGISTRY_NAME #eg: getting-started-fastai
\n–repository-format=docker
\n–location=REGION #eg: europe-west1<\/div>\n<\/div>- <\/i><\/span>\n
Build your docker image<\/p>\n<\/div><\/li><\/ul>
docker build -f TextDockerfile -t REGION-docker.pkg.dev\/PROJECT_ID\/ARTIFACT_REGISTRY_NAME\/fastai_text_cls:v0<\/div>\n<\/div>2\u20132 (Optional) Check that your docker image runs fine<\/h3><\/div>- <\/i><\/span>\n
Run your docker image locally and test it<\/p>\n<\/div><\/li><\/ul>
docker run -it -p 8080:8080 REGION-docker.pkg.dev\/PROJECT_ID\/ARTIFACT_REGISTRY_NAME\/fastai_text_cls:v0
\ncurl -X POST -H \"Content-Type: application\/json\" -d '[\"this was a bad movie\"]' 127.0.0.1:8080\/predictions\/fastai_model\n<\/div>\n<\/div>2\u20133 Push your docker image to a container registry in your GCP project<\/h3><\/div>You need to have the IAM credentials to do that. Once you\u2019ve ensured you have them, run the following<\/p>\n<\/div>
gcloud auth configure-docker
\ndocker push REGION-docker.pkg.dev\/PROJECT_ID\/ARTIFACT_REGISTRY_NAME\/fastai_text_cls:v0<\/div>\n<\/div>2\u20134 Create a model version using your docker image<\/h3><\/div>gcloud beta ai-platform versions create VERSION_NAME
\n–region=REGION
\n–model=MODEL_NAME
\n–image=REGION-docker.pkg.dev\/PROJECT_ID\/ARTIFACT_REGISTRY_NAME\/fastai_text_cls:v0
\n–ports=8080
\n–health-route=\/ping
\n–predict-route=\/predictions\/fastai_model<\/div>\n<\/div>2\u20135 Test your model version<\/h3><\/div>curl -X POST
\n-H \"Authorization: Bearer $(gcloud auth print-access-token)\"
\n-H \"Content-Type: application\/json\"
\n-d '[\"this was a bad movie\"]'
\nhttps:\/\/REGION-ml.googleapis.com\/v1\/projects\/PROJECT_ID\/models\/MODEL_NAME\/versions\/VERSION_NAME:predict\n<\/div>\n<\/div>Your fastai model is now deployed in a serverless architecture on AI Platform Prediction. You can make online predictions by sending requests to your model as a REST API. All methods to request predictions can be found in google documentation<\/a>.<\/p>\n<\/div>Conclusion<\/h2><\/div>Using AI Platform Prediction to serve any type of model can be very useful. This article was aimed to show an example of a deep learning model using a heavy framework (pytorch) and serve it in a cost effective way.<\/p>\n<\/div>
Some limitations are to keep in mind:<\/p>\n<\/div>
- <\/i><\/span>\n
Even with autoscaling, it is not possible to downscale to 0 instances when you use AI Platform models deployed on regional endpoints. Since that\u2019s the only option to use custom containers, you\u2019ll always have at least one instance up<\/p>\n<\/div><\/li>
- <\/i><\/span>\n
Another explored option was to use custom routines rather than custom containers but you can only do so if your model and packaged code are below a limit size of 500 MB which in our case was not possible to achieve.<\/p>\n<\/div><\/li><\/ul><\/div><\/div><\/div><\/article>
In this second article of the series of two, I will dive into the deployment and the serving of our models at scale. If you missed the first one about training a fastai model at scale on AI Platform Training, here is the link<\/a>.<\/p>\n<\/div> Serving a deep learning model can reveal several challenges among which:<\/p>\n<\/div> scaling resources on instances with or without accelerators (NVIDIA GPUs)<\/p>\n<\/div><\/li> cost efficiency<\/p>\n<\/div><\/li><\/ul> In this article, I will explain how I served a deep learning text classifier trained with the FastAI library following 2 main steps:<\/p>\n<\/div> Deploy fastai model using TorchServe<\/p>\n<\/div><\/li> Host serving on GCP AI Platform Prediction<\/p>\n<\/div><\/li><\/ul> All materials can be found in the g<\/a>ithub repository<\/a>. This repository was inspired by another project that aimed to deploy a fastai image classifier on AWS SageMaker Inference Endpoint here<\/a>.<\/p>\n<\/div> TorchServe makes it easy to deploy PyTorch models at scale in production environments. It removes the heavy lifting of developing your own client server architecture. The FastAI library is based on the PyTorch framework. It makes it possible to use this technology to serve fastai models by loading your fastai model as a pure pytorch object (remove fastai abstraction).<\/p>\n<\/div> To do that, you need to restore the FastAI learner from the export pickle from the last post, and save its model weights with PyTorch.<\/p>\n<\/div> Once you\u2019ve exported your pytorch weights, you need to rebuild the model structure to be able to load your weights into it. You might have to dig a little bit in fastai source code to find your implementation but luckily, In Jupyter notebook, one can investigate the source code by adding ?? in front of a function name.<\/p>\n<\/div> For text classifier, you can load a pure pytorch object by using the fastai get_text_classifier function<\/p>\n<\/div> Once you have obtained your pytorch pure model, you need to apply the same preprocessing that was used for training. FastAI has a very handy method .predict that can be applied to a text (simple string object), that naturally reproduces training preprocessing and therefore removes risk of training serving skew.<\/p>\n<\/div> In our case, we have to take this responsibility ourselves, since we need to get rid of fastai abstraction and work directly with PyTorch objects.<\/p>\n<\/div> In my example, I used a spacy tokenizer so I reproduced fastai preprocessing as shown below: As you can notice, the results I get using torch functions and learn.predict are the same because I managed to preserve the same preprocessing steps.<\/p>\n<\/div> In this section we deploy the PyTorch model to TorchServe. For installation, please refer to TorchServe Github Repository. Call the API and get the response. As shown in \/deployment\/handler.py, the TorchServe handler accepts data and context. In our example, we define another helper Python class with 4 instance methods to implement: initialize, preprocess, inference and postprocess.<\/p>\n<\/div> Now it\u2019s ready to setup and launch TorchServe.<\/p>\n<\/div> The first call would have longer latency due to model weights loading defined in initialize, but this will be mitigated from the second call onward.<\/p>\n<\/div> In this section we deploy the FastAI trained model with TorchServe in GCP AI Platform Prediction using a customized Docker image. For more details about GCP AI Platform Prediction routines using custom containers please refer to this article. Note that this option is only available if you use AI Platform Prediction with regional endpoints.<\/p>\n<\/div> Steps to deploy a fastai model on AI Platform Prediction:<\/p>\n<\/div> First, create an AI Platform Prediction model on a regional endpoint:<\/p>\n<\/div> Create a folder model\/ in the root of the repository<\/p>\n<\/div><\/li> Place your fastai model weights in model\/text\/ and name it fastai_cls_weights.pth<\/p>\n<\/div><\/li> Create an artifact repository<\/p>\n<\/div><\/li><\/ul> Build your docker image<\/p>\n<\/div><\/li><\/ul> Run your docker image locally and test it<\/p>\n<\/div><\/li><\/ul> You need to have the IAM credentials to do that. Once you\u2019ve ensured you have them, run the following<\/p>\n<\/div> Your fastai model is now deployed in a serverless architecture on AI Platform Prediction. You can make online predictions by sending requests to your model as a REST API. All methods to request predictions can be found in google documentation<\/a>.<\/p>\n<\/div> Using AI Platform Prediction to serve any type of model can be very useful. This article was aimed to show an example of a deep learning model using a heavy framework (pytorch) and serve it in a cost effective way.<\/p>\n<\/div> Some limitations are to keep in mind:<\/p>\n<\/div> Even with autoscaling, it is not possible to downscale to 0 instances when you use AI Platform models deployed on regional endpoints. Since that\u2019s the only option to use custom containers, you\u2019ll always have at least one instance up<\/p>\n<\/div><\/li> Another explored option was to use custom routines rather than custom containers but you can only do so if your model and packaged code are below a limit size of 500 MB which in our case was not possible to achieve.<\/p>\n<\/div><\/li><\/ul><\/div><\/div><\/div><\/article>1- Deploy fastai model using TorchServe<\/h2><\/div>
1\u20131 Export Model Weights from FastAI<\/h3><\/div>
\nfrom fastai.text import load_learner
\nfrom fastai.text.learner import get_c, get_text_vocab
\nlearn = load_learner(“fastai_cls.pkl”)
\nvocab_sz = len(_get_text_vocab(dls)) #dls is the dataloader you used for training
\nn_class = get_c(dls)
\nconfig = awd_lstm_clas_config.copy()
\ntorch.save(learn.model.state_dict(), “fastai_cls_weights.pth”)<\/div>\n<\/div>1\u20132 PyTorch Model from FastAI<\/h3><\/div>
\nfrom fastai.text.all import AWD_LSTM
\ntorch_pure_model = get_text_classifier(AWD_LSTM, vocab_sz, n_class, config=config)<\/div>\n<\/div>1\u20133 Reproduce fastai preprocessing steps<\/h3><\/div>
\npred_fastai = learn.predict(text)
\npred_fastai
\n>>(Category tensor(1), tensor(1), tensor([0.0036, 0.9964]))<\/div>\n<\/div>
\nimport torch<\/p>\n<\/div>
\nfrom fastai.text.core import Tokenizer, SpacyTokenizerfrom fastai.text.data import Numericalize
\nexample = “Hello, this is a test.”
\ntokenizer = Tokenizer(
\ntok=SpacyTokenizer(“en”)
\n)
\nnumericalizer = Numericalize(vocab=vocab)
\nexample_processed = numericalizer(tokenizer(example))
\nexample_processed
\n>>> tensor([ 4, 7, 26, 29, 16, 72, 69, 31])
\ninputs = example_processed.resize(1, len(example_processed))
\noutputs = model_torch.forward(inputs)[0]\npreds = torch.softmax(outputs, dim=-1) #You can use any activation function you need
\npreds
\n>>> tensor([[0.0036, 0.9964]], grad_fn=)<\/div>\n<\/div>1\u20134 Deploy your model via torchserve<\/h3><\/div>
\nOverall, there are mainly 3 steps to use TorchServe:<\/p>\n\n
\nIn order to archive the model, at least 2 files are needed in our case:<\/p>\n\n
Custom Handler<\/h3><\/div>
TorchServe in Action<\/h3><\/div>
Step 1: Archive the model PyTorch<\/h6><\/div>
\n\u2014-model-name=fastai_model
\n–version=1.0
\n–serialized-file=\/home\/model-server\/fastai_cls_weights.pth
\n\u2014- extra-files=\/home\/model-server\/config.py,\/home\/model-server\/vocab.json
\n–handler=\/home\/model-server\/handler.py
\n\u2014-export-path=\/home\/model-server\/model-store\/<\/div>\n<\/div>Step 2: Serve the Model<\/h6><\/div>
Step 3: Call API and Get the Response (here we use curl).<\/h6><\/div>
2- Deployment to AI Platform Prediction<\/h2><\/div>
\n–region=REGION #eg: europe-west1
\n–enable-logging
\n–enable-console-logging<\/div>\n<\/div>2\u20131 Build your docker image that will be used by your version<\/h3><\/div>
\n–repository-format=docker
\n–location=REGION #eg: europe-west1<\/div>\n<\/div>2\u20132 (Optional) Check that your docker image runs fine<\/h3><\/div>
\ncurl -X POST -H \"Content-Type: application\/json\" -d '[\"this was a bad movie\"]' 127.0.0.1:8080\/predictions\/fastai_model\n<\/div>\n<\/div>2\u20133 Push your docker image to a container registry in your GCP project<\/h3><\/div>
\ndocker push REGION-docker.pkg.dev\/PROJECT_ID\/ARTIFACT_REGISTRY_NAME\/fastai_text_cls:v0<\/div>\n<\/div>2\u20134 Create a model version using your docker image<\/h3><\/div>
\n–region=REGION
\n–model=MODEL_NAME
\n–image=REGION-docker.pkg.dev\/PROJECT_ID\/ARTIFACT_REGISTRY_NAME\/fastai_text_cls:v0
\n–ports=8080
\n–health-route=\/ping
\n–predict-route=\/predictions\/fastai_model<\/div>\n<\/div>2\u20135 Test your model version<\/h3><\/div>
\n-H \"Authorization: Bearer $(gcloud auth print-access-token)\"
\n-H \"Content-Type: application\/json\"
\n-d '[\"this was a bad movie\"]'
\nhttps:\/\/REGION-ml.googleapis.com\/v1\/projects\/PROJECT_ID\/models\/MODEL_NAME\/versions\/VERSION_NAME:predict\n<\/div>\n<\/div>Conclusion<\/h2><\/div>