K
Q

How to solve Error obtaining Endpoints for Service "default/frontend-svc": no object matching key "default/frontend-svc" in local store

11/9/2020

I know this question has been asked before, but the previous one doesn't apply to my case. I have 2 deployments and 2 services and I am using ingress resource and nginx ingress controller

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: frontend-ingress
    annotations: 
      kubernetes.io/ingress.class: nginx
spec:
    rules:
    - http:
        paths:
        - path: /api
          backend:
            serviceName: frontend-svc
            servicePort: 3000
        - path: /
          backend:
            serviceName: static-svc
            servicePort: 80

And the 2 services are defined below,

services.yaml

  apiVersion: v1
  kind: Service
  metadata:
      labels:
          app: frontend
      name: frontend-svc
  spec:
      ports:
      - port: 3000
        protocol: TCP
        targetPort: 3000
      selector:
          app: frontend
      type: ClusterIP

---

apiVersion: v1
kind: Service
metadata:
    labels:
        app: static
    name: static-svc
spec:
    ports:
    - port: 80
      protocol: TCP
      targetPort: 80
    selector:
        app: static
    type: ClusterIP

The deployments are shown below,

apiVersion: apps/v1
kind: Deployment
metadata:
    labels:
        app: frontend
    name: frontend
spec:
    replicas: 2
    selector:
        matchLabels:
            app: frontend
    template:
        metadata:
            labels:
                app: frontend
        spec:
            containers:
            - image: magalixcorp/sample-api:v1
              imagePullPolicy: IfNotPresent
              name: frontend
              env:
              - name: REDIS_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: redis-password
                    key: redis-password
              volumeMounts:
              - name: config-volume
                mountPath: /app/config.json
                subPath: config.json
            volumes:
                - name: config-volume
                  configMap:
                    name: app-config

---

apiVersion: apps/v1
kind: Deployment
metadata:
    labels:
        app: static
    name: static
spec:
    replicas: 2
    selector:
        matchLabels:
            app: static
    template:
        metadata:
            labels:
                app: static
        spec:
            containers:
            - image: magalixcorp/static:v1
              imagePullPolicy: IfNotPresent
              name: static

I have deployed an nginx controller using helm to AKS. The routes are not working at all. When I check the nginx-controller logs i get this

W1109 16:30:33.892443       6 controller.go:849] Error obtaining Endpoints for Service "default/frontend-svc": no object matching key "default/frontend-svc" in local store
W1109 16:30:33.892478       6 controller.go:849] Error obtaining Endpoints for Service "default/static-svc": no object matching key "default/static-svc" in local store
I1109 16:30:33.929583       6 main.go:112] "successfully validated configuration, accepting" ingress="frontend-ingress/default"
W1109 16:30:33.935904       6 controller.go:849] Error obtaining Endpoints for Service "default/frontend-svc": no object matching key "default/frontend-svc" in local store
W1109 16:30:33.935930       6 controller.go:849] Error obtaining Endpoints for Service "default/static-svc": no object matching key "default/static-svc" in local store
I1109 16:30:33.935973       6 controller.go:144] "Configuration changes detected, backend reload required"
I1109 16:30:33.936084       6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"frontend-ingress", UID:"efde0161-c03f-49d4-8f2b-38e2ceaaf8a7", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"7945", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1109 16:30:34.005219       6 controller.go:161] "Backend successfully reloaded"
I1109 16:30:34.005615       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"nginx", Name:"nginx-ingress-nginx-controller-68f45f49f6-slnv7", UID:"7880b0be-a1ff-450d-b1c1-4cee11722a6b", APIVersion:"v1", ResourceVersion:"3538", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1109 16:30:37.269565       6 controller.go:144] "Configuration changes detected, backend reload required"
I1109 16:30:37.342929       6 controller.go:161] "Backend successfully reloaded"
I1109 16:30:37.343677       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"nginx", Name:"nginx-ingress-nginx-controller-68f45f49f6-slnv7", UID:"7880b0be-a1ff-450d-b1c1-4cee11722a6b", APIVersion:"v1", ResourceVersion:"3538", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1109 16:31:01.926983       6 status.go:290] "updating Ingress status" namespace="default" ingress="frontend-ingress" currentValue=[] newValue=[{IP: Hostname:a596c83cb03464004bcced0767dd5aa9-1681194742.us-east-1.elb.amazonaws.com}]
I1109 16:31:01.937928       6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"frontend-ingress", UID:"efde0161-c03f-49d4-8f2b-38e2ceaaf8a7", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"8081", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
10.0.1.165 - - [09/Nov/2020:16:32:18 +0000] "GET / HTTP/1.1" 200 1313 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0" 396 0.002 [default-static-svc-80] [] 10.0.3.144:80 1313 0.000 200 750627c05c4ca98a2bcd60ae8bbb8828
10.0.1.165 - - [09/Nov/2020:16:32:19 +0000] "GET /js/script.js HTTP/1.1" 200 314 "http://a596c83cb03464004bcced0767dd5aa9-1681194742.us-east-1.elb.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0" 397 0.001 [default-static-svc-80] [] 10.0.1.175:80 314 0.000 200 af72c4fdc493a489064307872636b662
10.0.1.165 - - [09/Nov/2020:16:32:20 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "http://a596c83cb03464004bcced0767dd5aa9-1681194742.us-east-1.elb.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0" 407 0.002 [default-static-svc-80] [] 10.0.3.144:80 153 0.000 404 ba36f92120a41720842ba95e24b9975c
10.0.2.31 - - [09/Nov/2020:16:32:47 +0000] "GET /.env HTTP/1.1" 404 555 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" 231 0.001 [default-static-svc-80] [] 10.0.1.175:80 555 0.000 404 d8ec13faad1d46c13401b397e2d80ad0
10.0.2.31 - - [09/Nov/2020:16:32:48 +0000] "POST / HTTP/1.1" 405 559 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" 317 0.001 [default-static-svc-80] [] 10.0.1.175:80 559 0.000 405 567a4b2499fc816d72135445076bafda
10.0.1.165 - - [09/Nov/2020:16:32:53 +0000] "GET /api HTTP/1.1" 200 3 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0" 399 0.003 [default-frontend-svc-3000] [] 10.0.1.26:3000 3 0.000 200 8a92b9829d26dccb60f11a9c8537d831
10.0.3.231 - - [09/Nov/2020:16:34:37 +0000] "POST /api HTTP/1.1" 204 0 "-" "curl/7.64.1" 211 0.002 [default-frontend-svc-3000] [] 10.0.1.26:3000 0 0.000 204 cf837f8bc89c29f2d2b07ac317120220
10.0.2.31 - - [09/Nov/2020:16:34:55 +0000] "GET /api HTTP/1.1" 200 3 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0" 425 0.006 [default-frontend-svc-3000] [] 10.0.2.98:3000 3 0.004 200 a5757323ff32d8c4dbc56ef476798954

When I try <externalip>:3000/api either to POST or GET it's not working. What could be wrong. What is it that am doing wrong.

-- Shammir
kubernetes
kubernetes-ingress
nginx-ingress

Similar Questions

Kubernetes dashboard in CrashLoopbackOff: Error while initializing connection to Kubernetes apiserve
Expose each pod in a statefulset to the internet without a custom proxy
standard_init_linux.go:178: exec user process caused "exec format error" kubernetes
What is the linux version of docker for mac k8s mode?
PHP Local development - minikube or docker-compose?

1 Answer

11/9/2020

Try cleaning this up a bit. So create a deployment just for your frontend. So this file you would call frontend-depl.yml and then write it like this:

apiVersion: apps/v1
kind: Deployment
metadata:
    name: frontend-depl
spec:
    replicas: 1
    selector:
        matchLabels:
            app: frontend
    template:
        metadata:
            labels:
                app: frontend
        spec:
            containers:
              - name: frontend
                image: magalixcorp/sample-api:v1
---

apiVersion: v1
kind: Service
metadata:
    name: frontend-srv
spec:
    selector:
      app: frontend
    ports:
     - name: frontend
       protocol: TCP
       port: 3000
       targetPort: 3000

I am suggesting this to keep your yml files neat and tidy as this can all spin out of control real fast as you can tell and troubleshooting can be a pain.

Also I would go into your terminal and run:

$ kubectl get endpoints --namespace <namespace>
...
<namespace> <your-app-service>  <none> 21h

That is to look and see what endpoints you have.

The issue could also be your label selectors, which is why I offered a cleaner version of your yml file, at least the frontend-depl one, I would use the same pattern for your other yml files.

To learn more about label selectors: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors

-- Daniel
Source: StackOverflow