I followed the steps given in this post to deploy my tensorflow model for prediction using GPUs on Google Kubernetes Engine and Kubeflow. I have exposed the service as a load balancer by modifying the YAML file in this way where I changed the type from ClusterIP to LoadBalancer.

spec:
  clusterIP: A.B.C.D
  externalTrafficPolicy: Cluster
  ports:
  - name: grpc-tf-serving
    nodePort: 30098
    port: 9000
    protocol: TCP
    targetPort: 9000
  - name: http-tf-serving-proxy
    nodePort: 31399
    port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    app: my-model
  sessionAffinity: None
  type: LoadBalancer

The status changed to:

status:
  loadBalancer:
    ingress:
    - ip: W.X.Y.Z

Service specs (kubectl describe services my-model):

Name:                     my-model
Namespace:                default
Labels:                   app=my-model
                          app.kubernetes.io/deploy-manager=ksonnet
                          ksonnet.io/component=model2
Annotations:              getambassador.io/config:
                            ---
                            apiVersion: ambassador/v0
                            kind:  Mapping
                            name: tfserving-mapping-my-model-get
                            prefix: /models/my-model/
                            rewrite: /
                            method: GET
                            service: my-model.default:8000
                            ---
                            apiVersion: ambassador/v0
                            kind:  Mapping
                            name: tfserving-mapping-my-model-post
                            prefix: /models/my-model/
                            rewrite: /model/my-model:predict
                            method: POST
                            service: my-model.default:8000
                          ksonnet.io/managed:
                            {"pristine":"H4sIAAAAAAAA/7SRMY/UQAyFe35F5DpzCVweRcHW4QQBWKlQzQMhS/jZEckHmvGt9xplf+OZvfYjXRCgoIyz+/L8xsfgTR+5VxiEkA4vIYWfkQJgHDH+RAHhhYWNgpkB...
Selector:                 app=my-model
Type:                     LoadBalancer
IP:                       A.B.C.D
LoadBalancer Ingress:     W.X.Y.Z
Port:                     grpc-tf-serving  9000/TCP
TargetPort:               9000/TCP
NodePort:                 grpc-tf-serving  30098/TCP
Endpoints:                P.Q.R.S:9000
Port:                     http-tf-serving-proxy  8000/TCP
TargetPort:               8000/TCP
NodePort:                 http-tf-serving-proxy  31399/TCP
Endpoints:                R.Q.R.S:8000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Pods Specs (kubectl describe pods):

Name:           my-model-v1-bd6ccb757-qrwdv
Namespace:      default
Node:           gke-kuberflow-xyz-gpu-pool-5d4ebf17-56mf/SOME_IP
Start Time:     Mon, 18 Feb 2019 18:11:24 +0530
Labels:         app=my-model
                pod-template-hash=682776313
                version=v1
Annotations:    <none>
Status:         Running
IP:             P.Q.R.S
Controlled By:  ReplicaSet/my-model-v1-bd6ccb757
Containers:
  my-model:
    Container ID:  docker://d14e8261ddfe606393da2ee45badac0136cee98rwa5611c47ad85733ce5d2c925
    Image:         tensorflow/serving:1.11.1-gpu
    Image ID:      docker-pullable://tensorflow/serving@sha256:907d7db828b28ewer234d0b3ca10e2d66bcd8ef82c5cccea761fcd4f1190191d2f
    Port:          9000/TCP
    Host Port:     0/TCP
    Command:
      /usr/bin/tensorflow_model_server
    Args:
      --port=9000
      --model_name=my-model
      --model_base_path=gs://xyz_kuber_app-xyz-identification/export/
    State:          Running
      Started:      Mon, 18 Feb 2019 18:11:25 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:             4
      memory:          4Gi
      nvidia.com/gpu:  1
    Requests:
      cpu:             1
      memory:          1Gi
      nvidia.com/gpu:  1
    Environment:       <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-b6dpn (ro)
  my-model-http-proxy:
    Container ID:  docker://c98e06ad75f3456c353395e9ad2e2e3bcbf0b38cd2634074704439cd5ebf335d
    Image:         gcr.io/kubeflow-images-public/tf-model-server-http-proxy:v20180606-asdasda
    Image ID:      docker-pullable://gcr.io/kubeflow-images-public/tf-model-server-http-proxy@sha256:SHA
    Port:          8000/TCP
    Host Port:     0/TCP
    Command:
      python
      /usr/src/app/server.py
      --port=8000
      --rpc_port=9000
      --rpc_timeout=10.0
    State:          Running
      Started:      Mon, 18 Feb 2019 18:11:25 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:        500m
      memory:     500Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-b6dpn (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  default-token-b6dpn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-fsdf3
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
                 nvidia.com/gpu:NoSchedule
Events:          <none>

I used the command python predict.py --url=http://W.X.Y.Z:8000/model/my-model:predict to perform the prediction from the serving_script folder but I am getting the a 500 Internal server error as the response. What is going wrong here?

The code for prediction can be found here: https://github.com/kubeflow/examples/tree/master/object_detection/serving_script