K
Q

K8s Ingress, initiate ingress controller nginx error?

5/29/2018

I have two spring boot container, I want to setup ingress service. As document here says, ingress has two parts, one is controller, the other is resources.

My two resources are two containers: gearbox-rack-eureka-server and gearbox-rack-config-server. The difference is port so that ingress could route traffic by different ports. My yaml files are listed below:

eureka_pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: gearbox-rack-eureka-server
  labels:
    app: gearbox-rack-eureka-server
    purpose: platform_eureka_demo
spec:
  containers:
  - name:  gearbox-rack-eureka-server
    image: 192.168.1.229:5000/gearboxrack/gearbox-rack-eureka-server
    ports:
        - containerPort: 8761

eureka_svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: gearbox-rack-eureka-server
  labels:
    name: gearbox_rack_eureka_server
spec:
  selector:
    app: gearbox-rack-eureka-server
  type: NodePort
  ports:
    - port: 8761
      nodePort: 31501
      name: tcp

config_pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: gearbox-rack-config-server
  labels:
    app: gearbox-rack-config-server
    purpose: platform-demo
spec:
  containers:
  - name:  gearbox-rack-config-server
    image: 192.168.1.229:5000/gearboxrack/gearbox-rack-config-server
    ports:
    - containerPort: 8888
    env:
      - name: EUREKA_SERVER
        value: http://172.16.100.83:8761

config_svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: gearbox-rack-config-server
  labels:
    name: gearbox-rack-config-server
spec:
  selector:
    app: gearbox-rack-config-server
  type: NodePort
  ports:
    - port: 8888
      nodePort: 31502
      name: tcp

My ingress-nginx controller is mostly copied from the link above,

ingress_nginx_ctl.yaml:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
spec:
  type: LoadBalancer
  selector:
    app: ingress-nginx
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: https
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: ingress-nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: ingress-nginx
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: nginx:1.13.12
        name: ingress-nginx
        imagePullPolicy: Always
        ports:
          - name: http
            containerPort: 80
            protocol: TCP
          - name: https
            containerPort: 443
            protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend

I did following commands, they are successful.

kubectl apply -f eureka_pod.yaml
kubectl apply -f eureka_svc.yaml
kubectl apply -f config_pod.yaml
kubectl apply -f config_svc.yaml

Then I got error from execute kubectl apply -f ingress_nginx_ctl.yaml, the pod does not start, logs are listed below:

[root@master3 nginx-ingress-controller]# kubectl get pods
NAME                             READY     STATUS             RESTARTS   AGE
gearbox-rack-config-server       1/1       Running            0          39m
gearbox-rack-eureka-server       1/1       Running            0          40m
ingress-nginx-686c9975d5-7d464   0/1       CrashLoopBackOff   6          7m
[root@master3 nginx-ingress-controller]# kubectl logs -f ingress-nginx-686c9975d5-7d464
container_linux.go:247: starting container process caused "exec: \"/nginx-ingress-controller\": stat /nginx-ingress-controller: no such file or directory"

I created a directory /nginx-ingress-controller under root, and repeat the steps again, it still said same error. Does someone could point me the problem?

I put my ingress_nginx_res.yaml as follows for reference, it may have errors also.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: 172.16.100.83
    http:
      paths:
      - backend:
          serviceName: gearbox-rack-eureka-server
          servicePort: 8761
  - host: 172.16.100.83
    http:
      paths:
      - path:
        backend:
          serviceName: gearbox-rack-config-server
          servicePort: 8888

\==========================================

second edition

After change image link, The previous errors disappear, but still it has following permission problem:

[root@master3 ingress]# kubectl get pods
NAME                             READY     STATUS             RESTARTS   AGE
gearbox-rack-config-server       1/1       Running            0          15m
gearbox-rack-eureka-server       1/1       Running            0          15m
ingress-nginx-8679f9c8ff-5sxw7   0/1       CrashLoopBackOff   5          12m

The log message is as follows:

[root@master3 kube]# kubectl logs ingress-nginx-8679f9c8ff-5sxw7
W0530 07:54:22.290114       5 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0530 07:54:22.290374       5 main.go:158] Creating API client for https://10.96.0.1:443
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.15.0
  Build:      git-df61bd7
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------
I0530 07:54:22.298248       5 main.go:202] Running in Kubernetes Cluster version v1.9 (v1.9.2) - git (clean) commit 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b - platform linux/amd64
F0530 07:54:22.298610       5 main.go:80] ✖ It seems the cluster it is running with Authorization enabled (like RBAC) and there is no permissions for the ingress controller. Please check the configuration

It is RBAC problem. I check the install script which is downloaded from forum:

heapster-rbac.yaml:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: heapster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:heapster
subjects:
- kind: ServiceAccount
  name: heapster
  namespace: kube-system

One of related kubelet start argument is as follows: (I do not know whether it is relevant).

Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"

By which way, I could grant permission to ingress controller? Just put namespace kube-system to ingress_nginx_ctl.yaml?

\================================================================

Third edition

I put Kun Li's codes into ingress_nginx_role_rb.yaml, and run the following commands:

kubectl apply -f eureka_pod.yaml
kubectl apply -f eureka_svc.yaml
kubectl apply -f config_pod.yaml
kubectl apply -f config_svc.yaml
kubectl apply -f ingress_nginx_role_rb.yaml (just copy paste from Kun Li's answer)
kubectl apply -f nginx_default_backend.yaml
kubectl apply -f ingress_nginx_ctl.yaml

nginx_default_backend.yaml file is listed below:

kind: Service
apiVersion: v1
metadata:
  name: nginx-default-backend
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: http
  selector:
    app: nginx-default-backend
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: nginx-default-backend
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx-default-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        image: chenliujin/defaultbackend
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP

ingress_nginx_ctl.yaml is listed below:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
spec:
  type: LoadBalancer
  selector:
    app: ingress-nginx
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: https
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: ingress-nginx
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: ingress-nginx
    spec:
      terminationGracePeriodSeconds: 60
      serviceAccount: lb
      containers:
      - image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0
        name: ingress-nginx
        imagePullPolicy: Always
        ports:
          - name: http
            containerPort: 80
            protocol: TCP
          - name: https
            containerPort: 443
            protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend

From here, we could see service ingress-nginx namespace is default, not kube-system. But anyway, controller is up.

[root@master3 ingress]# kubectl get pods -n kube-system
NAME                                      READY     STATUS    RESTARTS   AGE
calico-etcd-cdn8z                         1/1       Running   0          11m
calico-kube-controllers-d554689d5-tzdq5   1/1       Running   0          11m
calico-node-dz4d6                         2/2       Running   1          11m
coredns-65dcdb4cf-h62bh                   1/1       Running   0          11m
etcd-master3                              1/1       Running   0          10m
heapster-5c448886d-swp58                  1/1       Running   0          11m
ingress-nginx-6ccc799fbc-hq2rm            1/1       Running   0          9m
kube-apiserver-master3                    1/1       Running   0          10m

ingress-nginx pod's namespace is kube-system (shown above), but its service's namespace is default.(shown below).

[root@master3 ingress]# kubectl get service
NAME                         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
gearbox-rack-config-server   NodePort       10.97.211.136   <none>        8888:31502/TCP               43m
gearbox-rack-eureka-server   NodePort       10.106.69.13    <none>        8761:31501/TCP               43m
ingress-nginx                LoadBalancer   10.105.114.64   <pending>     80:30646/TCP,443:31332/TCP   42m
kubernetes                   ClusterIP      10.96.0.1       <none>        443/TCP                      44m

as mentioned in the comments, expert's response help me to move forward.

-- user84592
kubernetes
kubernetes-ingress

Similar Questions

A Solution to Kubernetes pods stuck on Terminating
K8S RBAC in Docker 2.0 EE
Assign pod to workers without using deployment specific content like nodeSelector. Security constraints
Does Kubernetes installation always needs Docker?
Maintaining original source and destination headers for incoming UDP data in kubernetes cluster
Nginx ingress controller setting unexpected X-Forwarded-* headers
How can I add external node to EKS cluster?
Private helm repository failed to download chart

1 Answer

5/29/2018

For ingress-controller, image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0 should be used. And you need setup nginx-default-backend pod and service.

About RBAC, I think you need a seviceaccount to deploy your nginx-ingress-controller, with the following roles and bindings:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: lb
  namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-normal
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-minimal
  namespace: kube-system
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      - "ingress-controller-leader-dev"
      - "ingress-controller-leader-prod"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-minimal
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-minimal
subjects:
  - kind: ServiceAccount
    name: lb
    namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-normal
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-normal
subjects:
  - kind: ServiceAccount
    name: lb
    namespace: kube-system
-- Kun Li
Source: StackOverflow