how many models can be deployed in single node in azure kubernetes service?
Working on deployment of 170 ml models using ML studio and azure Kubernetes service which is referred on the below doc link "https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/machine-learning/how-to-deploy-azure-kubernetes-service.md".
We are training the model using python script with the custom environment and we are registering the ml model on the Azure ML services. Once we register the mode we are deploying it on the AKS by using the container images.
While deploying the ML model we are able to deploy up 10 to 11 models per pods for each Node in AKS. When we try to deploy the model on the same node we are getting deployment timeout error and we are getting the below error message.
For deploying the model in Azure Kubernetes Service using python language with below deployment configuration (auth_enabled = Flase, autoscale_enabled = Flase, cpu_cores = 0.6, memory_gb = 1, cpu_cores_limit = 2, memory_gb_limit = 2).
We also checked on the azure documentation and we could able to find any configuration or deployment setup for aks nodes.
Can you please provide us more clarification regarding "The number of models to be deployed is limited to 1,000 models per deployment (per container)" and Can you please give insight/feedback on how to increase the number of a ml model deployed in each node Azure Kubernetes Service. Thanks!
Similar Questions
1 Answer
The best practice for create one container in each POD. Because if one container is goes down or occur failure so it will all total down of that POD. Because container uses share storage or files in a PODS and it has assigned one public IP at pod level and deploy only one model per container.
The podsPerCore parameter limits the number of pods the node can run based on the number of processor cores on the node. For example, if podsPerCore is set to 10 on a node with 4 processor cores, the maximum number of pods allowed on the node is 40.
Question : Can you please give insight/feedback on how to increase the number of a ml model deployed in each node Azure Kubernetes Service?
Answer : To increase the number of a ML model deployment in each node to create max number of PODS in a node not to increase the container in a POD.
By default, there can be a maximum of 110 Pods per node, and each node in the cluster has allocated /24 range for its Pods. This results in 256 Pod IPs per node. By having approximately twice as many available IP addresses as possible Pods.
Reference: https://cloud.google.com/kubernetes-engine/docs/best-practices/scalability
follow the above suggestion because there might be reached resource usages of particular PODs under you are deploying ML model. Create another POD and deploy it there.