Nginx-Ingress for TCP entrypoint not working

10/25/2019

I am using Nginx as Kubernetes Ingress controller. After following this simple example I was able to setup this example

Now I am trying to setup TCP entrypoint for logstash with following config

Logstash :

apiVersion: v1
kind: Secret
metadata:
  name: logstash-secret
  namespace: kube-logging
type: Opaque
data:
  tls.crt: "<base64 encoded>" #For logstash.test.domain.com
  tls.key: "<base64 encoded>" #For logstash.test.domain.com

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-config
  namespace: kube-logging
  labels:
    app: logstash
data:
  syslog.conf: |-
    input {
      tcp {
          port => 5050
          type => syslog
      }
    }

    filter {
        grok {
          match => {"message" => "%{SYSLOGLINE}"}
        }
    }

    output {
      elasticsearch {
        hosts => ["http://elasticsearch:9200"] #elasticsearch running in same namespace (kube-logging)
        index => "syslog-%{+YYYY.MM.dd}"
      }
      stdout { codec => rubydebug }
    }

---

kind: Deployment
apiVersion: apps/v1
metadata:
  name: logstash
  namespace: kube-logging
  labels:
    app: logstash
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: logstash
    spec:
      #serviceAccountName: logstash
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:7.2.1
        imagePullPolicy: Always
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        ports:
        - name: logstash
          containerPort: 5050
          protocol: TCP
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: config
          mountPath: /usr/share/logstash/pipeline/syslog.conf
          readOnly: true
          subPath: syslog.conf
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: logstash-config

---

kind: Service
apiVersion: v1
metadata:
  name: logstash
  namespace: kube-logging
  labels:
    app: logstash
spec:
  ports:
    - name: tcp-port
      protocol: TCP
      port: 5050
      targetPort: 5050
  selector:
    app: logstash

Nginx-Ingress:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress
  namespace: kube-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
    spec:
      serviceAccountName: nginx-ingress
      containers:
      - image: nginx/nginx-ingress:1.5.7
        imagePullPolicy: Always
        name: nginx-ingress
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: tcp5050
          containerPort: 5050
        securityContext:
          allowPrivilegeEscalation: true
          runAsUser: 101 #nginx
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        args:
          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
          - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
          - -v=3 # Enables extensive logging. Useful for troubleshooting.
         #- -report-ingress-status
         #- -external-service=nginx-ingress
         #- -enable-leader-election
         #- -enable-prometheus-metrics
         #- -enable-custom-resources

LoadBalancer :

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-external
  namespace: kube-ingress
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80
  - name: https
    protocol: TCP
    port: 443
    targetPort: 443
  - name: tcp5050
    protocol: TCP
    port: 5050
    targetPort: 5050
  selector:
    app: nginx-ingress

Ingress :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: logstash-ingress
  namespace: kube-logging
spec:
  tls:
  - hosts:
    - logstash.test.domain.com
    secretName: logstash-secret #This has self-signed cert for logstash.test.domain.com
  rules:
  - host: logstash.test.domain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: logstash
          servicePort: 5050

With this config it shows following ,

NAME               HOSTS                         ADDRESS   PORTS     AGE
logstash-ingress   logstash.test.domain.com                 80, 443   79m

Why port 5050 not listed here ?

Just want to expose logstash service through public endpoint. When I use openssl s_client -connect logstash.kube-logging.svc.cluster.local:5050 within the cluster I get

$ openssl s_client -connect logstash.kube-logging.svc.cluster.local:5050
CONNECTED(00000005)

But from outside of the cluster openssl s_client -connect logstash.test.domain.com:5050 I get

$ openssl s_client -connect logstash.test.domain.com:5050
connect: Connection refused
connect:errno=61

and

$ openssl s_client -cert logstash_test_domain_com.crt -key logstash_test_domain_com.key -servername logstash.test.domain.com:5050
connect: Connection refused
connect:errno=61

What I need to do to get this working ?

-- roy
kubernetes-ingress
logstash
nginx-ingress

1 Answer

10/27/2019

It seems like you are kind of confused. So let's start by ordering your services and ingress.

First, there are 3 types of services in kubernetes. ClusterIP which allows you to expose your deployments internally in k8s. Nodeport which is the same as ClusterIP but also exposes your deployment through every node external IP and a PORT which is in the range ~30K-32K. Finally there is the LoadBalancer service which is the same as ClusterIP but also exposes your app in a specific external IP address assigned by the cloud provider LoadBalancer.

The NodePort service you created will make logstash accessible through every node external IP in a random port in the range 30K to 32K; find the port running kubectl get services | grep nginx-ingress and check the last column. To get your node's external ip addresses run kubectl get node -o wide. The LoadBalancer service you created will make logstash accessible through an external IP address in the port 5050. To find out the IP run kubectl get services | grep nginx-ingress-external. Finally, you have also created an ingress resource to reach logstash. For this you have defined a host which given TLS will be accessible in port 443 and inbound traffic there will be redirected to logstash's service type ClusterIP in port 5050. So there you go you have 3 ways to reach logstash. I would go for the LoadBalancer given that it is a specific port.

-- Rodrigo Loza
Source: StackOverflow