K
Q

I am trying to use externalname together with nodeport service for Nginx-Controller but I am getting 502 Bad Gateway

3/25/2019

Environment:

I have:

1- NGINX Ingress controller version: 1.15.9, image: 0.23.0

2- Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: Virtual Machines on KVM

OS (e.g. from /etc/os-release):

NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

Kernel (e.g. uname -a):

Linux node01 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Install tools: kubeadm

More details:

CNI : WEAVE

Setup:

  1. 2 Resilient HA Proxy, 3 Masters, 2 infra, and worker nodes.
  2. I am exposing all the services as node ports, where the HA-Proxy re-assign them to a public virtual IP.
  3. Dedicated project hosted on the infra node carrying the monitoring and logging tools (Grafana, Prometheus, EFK, etc)
  4. Backend NFS storage as persistent storage

What happened: I want to be able to use external Name rather than node ports, so instead of accessing grafana for instance via vip + 3000 I want to access it via http://grafana.wild-card-dns-zone

Deployment

  1. I have created a new namespace called ingress
  2. I deployed it as follow:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: **2**
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
      name: nginx-ingress
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        node-role.kubernetes.io/infra: infra
      terminationGracePeriodSeconds: 60
      containers:
      - name: nginx-ingress-controller
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        args:
          - /nginx-ingress-controller
          - --default-backend-service=ingress/ingress-controller-nginx-ingress-default-backend
          - --configmap=$(POD_NAMESPACE)/nginx-configuration
          - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
          - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          - --publish-service=$(POD_NAMESPACE)/ingress-nginx
          - --annotations-prefix=nginx.ingress.kubernetes.io
          - --v3
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            drop:
              - ALL
            add:
              - NET_BIND_SERVICE
          # www-data -> 33
          runAsUser: 33
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  generation: 1
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.3.1
    component: default-backend
  name: ingress-controller-nginx-ingress-default-backend
  namespace: ingress
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx-ingress
      component: default-backend
      release: ingress-controller
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx-ingress
        component: default-backend
        release: ingress-controller
    spec:
      nodeSelector:
        node-role.kubernetes.io/infra: infra
      containers:
      - image: k8s.gcr.io/defaultbackend:1.4
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: nginx-ingress-default-backend
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 60
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    name: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.3.1
    component: default-backend
  name: ingress-controller-nginx-ingress-default-backend
  namespace: ingress
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: nginx-ingress
    component: default-backend
    release: ingress-controller
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrolebinding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-rolebinding
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress

INGRESS SETUP:

services

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2019-03-25T16:03:01Z"
  labels:
    app: jaeger
    app.kubernetes.io/component: query
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  name: jeager-query
  namespace: monitoring-logging
  resourceVersion: "3055947"
  selfLink: /api/v1/namespaces/monitoring-logging/services/jeager-query
  uid: 778550f0-4f17-11e9-9078-001a4a16021e
spec:
  externalName: jaeger.example.com
  ports:
  - port: 16686
    protocol: TCP
    targetPort: 16686
  selector:
    app: jaeger
    app.kubernetes.io/component: query
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2019-03-25T15:40:30Z"
  labels:
    app: grafana
    chart: grafana-2.2.4
    heritage: Tiller
    release: grafana
  name: grafana
  namespace: monitoring-logging
  resourceVersion: "3053698"
  selfLink: /api/v1/namespaces/monitoring-logging/services/grafana
  uid: 51b9d878-4f14-11e9-9078-001a4a16021e
spec:
  externalName: grafana.example.com
  ports:
  - name: http
    port: 3000
    protocol: TCP
    targetPort: 3000
  selector:
    app: grafana
    release: grafana
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}

INGRESS

Ingress 1

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/service-upstream: "true"
  creationTimestamp: "2019-03-25T21:13:56Z"
  generation: 1
  labels:
    app: jaeger
    app.kubernetes.io/component: query-ingress
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  name: jaeger-query
  namespace: monitoring-logging
  resourceVersion: "3111683"
  selfLink: /apis/extensions/v1beta1/namespaces/monitoring-logging/ingresses/jaeger-query
  uid: e6347f6b-4f42-11e9-9e8e-001a4a16021c
spec:
  rules:
  - host: jaeger.example.com
    http:
      paths:
      - backend:
          serviceName: jeager-query
          servicePort: 16686
status:
  loadBalancer: {}

Ingress 2

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"labels":{"app":"grafana"},"name":"grafana","namespace":"monitoring-logging"},"spec":{"rules":[{"host":"grafana.example.com","http":{"paths":[{"backend":{"serviceName":"grafana","servicePort":3000}}]}}]}}
  creationTimestamp: "2019-03-25T17:52:40Z"
  generation: 1
  labels:
    app: grafana
  name: grafana
  namespace: monitoring-logging
  resourceVersion: "3071719"
  selfLink: /apis/extensions/v1beta1/namespaces/monitoring-logging/ingresses/grafana
  uid: c89d7f34-4f26-11e9-8c10-001a4a16021d
spec:
  rules:
  - host: grafana.example.com
    http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: 3000
status:
  loadBalancer: {}

EndPoints

Endpoint 1

apiVersion: v1
kind: Endpoints
metadata:
  creationTimestamp: "2019-03-25T15:40:30Z"
  labels:
    app: grafana
    chart: grafana-2.2.4
    heritage: Tiller
    release: grafana
  name: grafana
  namespace: monitoring-logging
  resourceVersion: "3050562"
  selfLink: /api/v1/namespaces/monitoring-logging/endpoints/grafana
  uid: 51bb1f9c-4f14-11e9-9e8e-001a4a16021c
subsets:
- addresses:
  - ip: 10.42.0.15
    nodeName: kuinfra01.example.com
    targetRef:
      kind: Pod
      name: grafana-b44b4f867-bcq2x
      namespace: monitoring-logging
      resourceVersion: "1386975"
      uid: 433e3d21-4827-11e9-9e8e-001a4a16021c
  ports:
  - name: http
    port: 3000
    protocol: TCP

Endpoint 2

apiVersion: v1
kind: Endpoints
metadata:
  creationTimestamp: "2019-03-25T16:03:01Z"
  labels:
    app: jaeger
    app.kubernetes.io/component: service-query
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  name: jeager-query
  namespace: monitoring-logging
  resourceVersion: "3114702"
  selfLink: /api/v1/namespaces/monitoring-logging/endpoints/jeager-query
  uid: 7786d833-4f17-11e9-9e8e-001a4a16021c
subsets:
- addresses:
  - ip: 10.35.0.3
    nodeName: kunode02.example.com
    targetRef:
      kind: Pod
      name: jeager-query-7d9775d8f7-2hwdn
      namespace: monitoring-logging
      resourceVersion: "3114693"
      uid: fdac9771-4f49-11e9-9e8e-001a4a16021c
  ports:
  - name: query
    port: 16686
    protocol: TCP

I am able to curl the endpoints from inside the ingress-controller pod:

# kubectl exec -it nginx-ingress-controller-5dd67f88cc-z2g8s  -n ingress -- /bin/bash
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ curl -k https://localhost
<a href="/login">Found</a>.

www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ curl http://localhost
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.15.9</center>
</body>
</html>
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ exit

But from out side when I am trying to reach jaeger.example.com or grafana.example.com I am getting 502 bad gatway and the following error log:

10.39.0.0 - [10.39.0.0] - - [25/Mar/2019:16:40:32 +0000] "GET /search HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" 514 0.001 [monitoring-logging-jeager-query-16686] vip:16686, vip:16686, vip:16686 0, 0, 0 0.001, 0.000, 0.000 502, 502, 502 b7c813286fccf27fffa03eb6564edfd1
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
10.39.0.0 - [10.39.0.0] - - [25/Mar/2019:16:40:32 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://jeager.example.com/search" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" 494 0.001 [monitoring-logging-jeager-query-16686] vip:16686, vip:16686, vip:16686 0, 0, 0 0.000, 0.001, 0.000 502, 502, 502 9e582912614e67dfee6be1f679de5933
I0325 16:40:32.497868       8 socket.go:225] skiping metric for host jeager.example.com that is not being served
I0325 16:40:32.497886       8 socket.go:225] skiping metric for host jeager.example.com that is not being served
-- A Hassan
kubernetes
kubernetes-ingress

Similar Questions

Intermittent failure of container mounts in Kubernetes
Get latest version in Chart.yaml
Convert YAML string to dict in Helm template
Does the Bolt protocol work with Traefik?
AKS cluster creation fails with error &#39;Security rule has invalid Port range&#39;
AWS EKS - Load Balancer (Ingress)
kubernetes - failed to sync configmap cache: timed out waiting for the condition

1 Answer

3/28/2019

First thanks for cookiedough for the clue to help regarding the service issue, but later I faced an issue to create service using external name but I found my mistake thanks for "Long" user in the slack, the mistake is that I was using service of type ExternalName and it should be type cluster IP here are the steps to solve the problems (Remark https issue is a separate problem): 1- Create wild character DNS zone pointing the public IP 1- For new service just create it of type ClusterIP 2- In the namespace for the service create an ingress using the following example (yaml):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
labels:
app: grafana
name: grafana
namespace: grafana-namespace
spec:
rules:

host: grafana.example.com
http:
paths:
backend:
serviceName: grafana
servicePort: 3000

3- kubectl -f apply -f grafana-ingress.yaml Now you can reach your grafana on http://grafana.example,com

-- A Hassan
Source: StackOverflow