Kubernetes - Ingress: Error: connect ECONNREFUSED 127.0.0.1:80

7/13/2020

I am trying to run ingress on minikube. I am running on Ubnutu 18.04. I am interested in nginx-ingress from: https://kubernetes.github.io/ingress-nginx/deploy/

I have simple test service which is running on port 3000 inside docker container. This container is pushed to docker hub. I have simple get request there:

app.get('/api/users/currentuser', (req, res) => {
  res.send('Hi there!');
});

Steps I've done: minikube start then minikube addons enable ingress after that I got message from minikube:

🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

but still when I am trying to verify if it is running I don't think that this is working fine: Output from kubectl get pods -n kube-system

NAME                                        READY   STATUS      RESTARTS   AGE
coredns-66bff467f8-bhqnk                    1/1     Running     4          2d8h
etcd-minikube                               1/1     Running     4          2d8h
ingress-nginx-admission-create-676jc        0/1     Completed   0          168m
ingress-nginx-admission-patch-bwf7x         0/1     Completed   0          168m
ingress-nginx-controller-7bb4c67d67-x5qzl   1/1     Running     3          168m
kube-apiserver-minikube                     1/1     Running     4          2d8h
kube-controller-manager-minikube            1/1     Running     4          2d8h
kube-proxy-jg2jz                            1/1     Running     4          2d8h
kube-scheduler-minikube                     1/1     Running     4          2d8h
storage-provisioner                         1/1     Running     6          2d8h

There is also no information about ingress when doing kubectl get pods

NAME                         READY   STATUS    RESTARTS   AGE
auth-depl-7dff4bb675-bpzfh   1/1     Running   0          4m58s

I am running services via skaffold using command skaffold dev

configuration of ingress service looks like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
  rules:
    - host: kube-test.dev
      http:
        paths:
          - path: /api/users/?(.*)
            backend:
              serviceName: auth-srv
              servicePort: 3000

which points to deployment file of (host of kube-test.dev is just mapped to localhost in /etc/hosts):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: auth
  template:
    metadata:
      labels:
        app: auth
    spec:
      containers:
        - name: auth
          image: geborskimateusz/auth
---
apiVersion: v1
kind: Service
metadata:
  name: auth-srv
spec:
  selector:
    app: auth
  ports:
    - name: auth
      protocol: TCP
      port: 3000
      targetPort: 3000

what is more if this does matter skaffold config looks like:

apiVersion: skaffold/v2alpha3
kind: Config
deploy:
  kubectl:
    manifests:
      - ./infra/k8s/*
build:
  local:
    push: false
  artifacts:
    - image: geborskimateusz/auth
      context: auth
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .

Any ideas here? I was running similar config on mac and it worked fine, this looks more like ingress addon problem. Any ideas here?

when I hit kube-test.dev/api/users/currentuser I get:

Error: connect ECONNREFUSED 127.0.0.1:80

and hosts file:

127.0.0.1       localhost
127.0.1.1       mat-5474

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

127.0.0.1 kube-test.dev
                    

EDIT

kubectl describe pods auth-depl-6845657cbc-kqdm8

Name:         auth-depl-6845657cbc-kqdm8
Namespace:    default
Priority:     0
Node:         minikube/192.168.99.100
Start Time:   Tue, 14 Jul 2020 09:51:03 +0200
Labels:       app=auth
              app.kubernetes.io/managed-by=skaffold-v1.12.0
              pod-template-hash=6845657cbc
              skaffold.dev/builder=local
              skaffold.dev/cleanup=true
              skaffold.dev/deployer=kubectl
              skaffold.dev/docker-api-version=1.40
              skaffold.dev/run-id=fcdee662-da9c-48ab-aab0-a6ed0ecef301
              skaffold.dev/tag-policy=git-commit
              skaffold.dev/tail=true
Annotations:  <none>
Status:       Running
IP:           172.17.0.4
IPs:
  IP:           172.17.0.4
Controlled By:  ReplicaSet/auth-depl-6845657cbc
Containers:
  auth:
    Container ID:   docker://674d4aae381431ff124c8533250a6206d044630135854e43ac70f2830764ce0a
    Image:          geborskimateusz/auth:2d55de4779465ed71686bffc403e6ad7cfef717e7d297ec90ef50a363dc5d3c7
    Image ID:       docker://sha256:2d55de4779465ed71686bffc403e6ad7cfef717e7d297ec90ef50a363dc5d3c7
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 14 Jul 2020 09:51:04 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-pcj8j (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-pcj8j:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-pcj8j
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age        From               Message
  ----    ------     ----       ----               -------
  Normal  Scheduled  <unknown>  default-scheduler  Successfully assigned default/auth-depl-6845657cbc-kqdm8 to minikube
  Normal  Pulled     47s        kubelet, minikube  Container image "geborskimateusz/auth:2d55de4779465ed71686bffc403e6ad7cfef717e7d297ec90ef50a363dc5d3c7" already present on machine
  Normal  Created    47s        kubelet, minikube  Created container auth
  Normal  Started    47s        kubelet, minikube  Started container auth

EDIT 2 kubectl logs ingress-nginx-controller-7bb4c67d67-x5qzl -n kube-system

W0714 07:49:38.776541       6 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0714 07:49:38.776617       6 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0714 07:49:38.777097       6 main.go:220] Creating API client for https://10.96.0.1:443
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       0.32.0
  Build:         git-446845114
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.17.10

-------------------------------------------------------------------------------

I0714 07:49:38.791783       6 main.go:264] Running in Kubernetes cluster version v1.18 (v1.18.3) - git (clean) commit 2e7996e3e2712684bc73f0dec0200d64eec7fe40 - platform linux/amd64
I0714 07:49:39.007305       6 main.go:105] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I0714 07:49:39.008092       6 main.go:113] Enabling new Ingress features available since Kubernetes v1.18
W0714 07:49:39.010806       6 main.go:125] No IngressClass resource with name nginx found. Only annotation will be used.
I0714 07:49:39.022204       6 ssl.go:528] loading tls certificate from certificate path /usr/local/certificates/cert and key path /usr/local/certificates/key
I0714 07:49:39.058275       6 nginx.go:263] Starting NGINX Ingress controller
I0714 07:49:39.076400       6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"nginx-load-balancer-conf", UID:"3af0b029-24c9-4033-8d2a-de7a15b62464", APIVersion:"v1", ResourceVersion:"2007", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap kube-system/nginx-load-balancer-conf
I0714 07:49:39.076438       6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"tcp-services", UID:"bbd76f82-e3b3-42f8-8098-54a87beb34fe", APIVersion:"v1", ResourceVersion:"2008", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap kube-system/tcp-services
I0714 07:49:39.076447       6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"udp-services", UID:"21710ee0-4b23-4669-b265-8bf5be662871", APIVersion:"v1", ResourceVersion:"2009", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap kube-system/udp-services
I0714 07:49:40.260006       6 nginx.go:307] Starting NGINX process
I0714 07:49:40.261693       6 leaderelection.go:242] attempting to acquire leader lease  kube-system/ingress-controller-leader-nginx...
I0714 07:49:40.262598       6 nginx.go:327] Starting validation webhook on :8443 with keys /usr/local/certificates/cert /usr/local/certificates/key
I0714 07:49:40.262974       6 controller.go:139] Configuration changes detected, backend reload required.
I0714 07:49:40.302595       6 leaderelection.go:252] successfully acquired lease kube-system/ingress-controller-leader-nginx
I0714 07:49:40.304129       6 status.go:86] new leader elected: ingress-nginx-controller-7bb4c67d67-x5qzl
I0714 07:49:40.437999       6 controller.go:155] Backend successfully reloaded.
I0714 07:49:40.438145       6 controller.go:164] Initial sync, sleeping for 1 second.
W0714 07:51:03.723044       6 controller.go:909] Service "default/auth-srv" does not have any active Endpoint.
I0714 07:51:03.765397       6 main.go:115] successfully validated configuration, accepting ingress ingress-service in namespace default
I0714 07:51:03.771212       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-service", UID:"96ec6e26-2354-46c9-be45-ca17a5f1a6f3", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"3991", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/ingress-service
I0714 07:51:07.032427       6 controller.go:139] Configuration changes detected, backend reload required.
I0714 07:51:07.115511       6 controller.go:155] Backend successfully reloaded.
I0714 07:51:40.319830       6 status.go:275] updating Ingress default/ingress-service status from [] to [{192.168.99.100 }]
I0714 07:51:40.332044       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-service", UID:"96ec6e26-2354-46c9-be45-ca17a5f1a6f3", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"4011", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/ingress-service
W0714 07:55:28.215453       6 controller.go:822] Error obtaining Endpoints for Service "default/auth-srv": no object matching key "default/auth-srv" in local store
I0714 07:55:28.215542       6 controller.go:139] Configuration changes detected, backend reload required.
I0714 07:55:28.234472       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-service", UID:"96ec6e26-2354-46c9-be45-ca17a5f1a6f3", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"4095", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/ingress-service
I0714 07:55:28.297582       6 controller.go:155] Backend successfully reloaded.
I0714 07:55:31.549294       6 controller.go:139] Configuration changes detected, backend reload required.
I0714 07:55:31.653169       6 controller.go:155] Backend successfully reloaded.
W0714 08:25:53.145312       6 controller.go:909] Service "default/auth-srv" does not have any active Endpoint.
I0714 08:25:53.188326       6 main.go:115] successfully validated configuration, accepting ingress ingress-service in namespace default
I0714 08:25:53.191134       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-service", UID:"4ac33fc5-ae7a-4511-922f-7e6bdc1fe4d5", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"4124", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/ingress-service
I0714 08:25:54.270931       6 status.go:275] updating Ingress default/ingress-service status from [] to [{192.168.99.100 }]
I0714 08:25:54.278468       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-service", UID:"4ac33fc5-ae7a-4511-922f-7e6bdc1fe4d5", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"4136", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/ingress-service
I0714 08:25:56.460808       6 controller.go:139] Configuration changes detected, backend reload required.
I0714 08:25:56.530559       6 controller.go:155] Backend successfully reloaded.

kubectl describe svc auth-srv

Name:              auth-srv
Namespace:         default
Labels:            app.kubernetes.io/managed-by=skaffold-v1.12.0
                   skaffold.dev/builder=local
                   skaffold.dev/cleanup=true
                   skaffold.dev/deployer=kubectl
                   skaffold.dev/docker-api-version=1.40
                   skaffold.dev/run-id=19ab20fe-baa5-4faf-a478-c1bad98a22b1
                   skaffold.dev/tag-policy=git-commit
                   skaffold.dev/tail=true
Annotations:       Selector:  app=auth
Type:              ClusterIP
IP:                10.99.52.173
Port:              auth  3000/TCP
TargetPort:        3000/TCP
Endpoints:         172.17.0.4:3000
Session Affinity:  None
Events:            <none>
-- Mateusz Gebroski
kubernetes
nginx-ingress
skaffold

3 Answers

7/17/2020

You've mentioned that you are (1) using minikube on Ubuntu, and (2) launching with minikube start. So it's almost certain that minikube is launching a VM (VirtualBox) to run Kubernetes, and thus your containers are not available through localhost.

There are a few ways to access your applications running within minikube. Minikube has a few options of its own. You can try proxying connections to your local box with kubectl proxy. AndSkaffold also supports port-forwarding individual services, etc. too. I'm actually not sure which is the best to use with Istio.

-- Brian de Alwis
Source: StackOverflow

9/29/2021

Please check the microk8s documentation page

you need to enable/install ingress-nginx

-- user2430821
Source: StackOverflow

7/17/2020

run minikube ip. You will get an ip address, copy paste in etc/hosts file for kube-test.dev

-- Yash Srivastava
Source: StackOverflow