As Michael says, quite possibly your labels or namespaces are mismatching. However in addition to that, keep in mind that even when you fix the endpoint, the url you're after (http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana) might not work correctly.

Depending on your root_url and/or static_root_path grafana configuration settings, when trying to login you might get grafana trying to POST to http://localhost:8001/login and get a 404.

Try using kubectl port-forward instead:

kubectl -n monitoring port-forward [grafana-pod-name] 3000

then access grafana via http://localhost:3000/

https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/

There are a few factors that might be causing this issue.

1. The service expects to find one or more supporting endpoints, which it discovers through matching rules on the labels. If the labels don't align, then the service won't find endpoints, and the network gateway function performed by the service will result in 503.

2. The port declared by the POD and the process within the container are misaligned from the --target-port expected by the service.

Either one of these might generate the error. Let's take a closer look.

First, kubectl describe the service:

$ kubectl describe svc grafana01-grafana-3000
Name:           grafana01-grafana-3000
Namespace:      default
Labels:         app=grafana01-grafana
            chart=grafana-0.3.7
            component=grafana
            heritage=Tiller
            release=grafana01
Annotations:        <none>
Selector:       app=grafana01-grafana,component=grafana,release=grafana01
Type:           NodePort
IP:         10.0.0.197
Port:           <unset> 3000/TCP
NodePort:       <unset> 30905/TCP
Endpoints:      10.1.45.69:3000
Session Affinity:   None
Events:         <none>

Notice that my grafana service has 1 endpoint listed (there could be multiple). The error above in your example indicates that you won't have endpoints listed here.

Endpoints:      10.1.45.69:3000

Let's take a look next at the selectors. In the example above, you can see I have 3 selector labels on my service:

Selector:       app=grafana01-grafana,component=grafana,release=grafana01

I'll kubectl describe my pods next:

$ kubectl describe pod grafana
Name:       grafana01-grafana-1843344063-vp30d
Namespace:  default
Node:       10.10.25.220/10.10.25.220
Start Time: Fri, 14 Jul 2017 03:25:11 +0000
Labels:     app=grafana01-grafana
        component=grafana
        pod-template-hash=1843344063
        release=grafana01
...

Notice that the labels on the pod align correctly, hence my service finds pods which provide endpoints which are load balanced against by the service. Verify that this part of the chain isn't broken in your environment.

If you do find that the labels are correct, you may still have a disconnect in that the grafana process running within the container within the pod is running on a different port than you expect.

$ kubectl describe pod grafana
Name:       grafana01-grafana-1843344063-vp30d
...
Containers:
  grafana:
    Container ID:   docker://69f11b7828c01c5c3b395c008d88e8640c5606f4d865107bf4b433628cc36c76
    Image:      grafana/grafana:latest
    Image ID:       docker-pullable://grafana/grafana@sha256:11690015c430f2b08955e28c0e8ce7ce1c5883edfc521b68f3fb288e85578d26
    Port:       3000/TCP
    State:      Running
      Started:      Fri, 14 Jul 2017 03:25:26 +0000

If for some reason, your port under the container listed a different value, then the service is effectively load balancing against an invalid endpoint.

For example, if it listed port 80: Port: 80/TCP Or was an empty value Port:

Then even if your label selectors were correct, the service would never find a valid response from the pod and would remove the endpoint from the rotation.

I suspect your issue is the first problem above (mismatched label selectors).

If both the label selectors and ports align, then you might have a problem with the MTU setting between nodes. In some cases, if the MTU used by your networking layer (like calico) is larger than the MTU of the supporting network, then you'll never get a valid response from the endpoint. Typically, this last potential issue will manifest itself as a timeout rather than a 503 though.

With Kubernetes 1.10 the proxy URL should be slighly different, like this:

http://localhost:8080/api/v1/namespaces/default/services/SERVICE-NAME:PORT-NAME/proxy/ 

Ref: https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#manually-constructing-apiserver-proxy-urls

Your Deployment may not have a label app: grafana, or be in another namespace. Could you also post the Deployment definition?