Kubernetes Ingress 502

12/2/2020

I setup a k8s cluster on my server(all in one node) and installed ingress-nginx. Now I want to deploy a frps service to the cluster. Here is my code:

apiVersion: v1
kind: Namespace
metadata:
  name: helloworld
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: frps-configmap
  namespace: helloworld
data:
  frps.ini: |
    [common]
    bind_port = 7000
    vhost_http_port = 8080
    custom_404_page = /etc/frp/frps.html
  frps.html: |
    <!DOCTYPE html>
    <html lang="en">
      // ...
    </html>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frps-deployment
  namespace: helloworld
  labels:
    app: frps
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frps
  template:
    metadata:
      labels:
        app: frps
    spec:
      containers:
        - name: frps
          image: 'snowdreamtech/frps:0.34.3'
          ports:
            - containerPort: 8080
            - containerPort: 7000
          volumeMounts:
            - name: frps-etc
              mountPath: '/etc/frp'
              readOnly: true
      volumes:
        - name: frps-etc
          configMap:
            name: frps-configmap
            items:
              - key: frps.ini
                path: frps.ini
              - key: frps.html
                path: frps.html
---
apiVersion: v1
kind: Service
metadata:
  name: frps
  namespace: helloworld
spec:
  selector:
    app: frps
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      name: vhost
    - protocol: TCP
      port: 7000
      targetPort: 7000
      name: frps
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frps-ingress
  namespace: helloworld
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: 'nginx'
  rules:
    - host: dev.helloworld.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frps
                port: 
                  name: vhost

And here are some cluster information after apply previous yaml:

kubectl get deploy -A
NAMESPACE         NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
helloworld        frps-deployment            1/1     1            1           12m
ingress-nginx     ingress-nginx-controller   1/1     1            1           23d
...

kubectl get svc -A
NAMESPACE       NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
helloworld      frps                                 ClusterIP   10.97.67.98      <none>        8080/TCP,7000/TCP            6m18s
ingress-nginx   ingress-nginx-controller             NodePort    10.101.128.200   <none>        80:32038/TCP,443:31363/TCP   23d
ingress-nginx   ingress-nginx-controller-admission   ClusterIP   10.97.36.119     <none>        443/TCP                      23d
...

kubectl describe ingress --namespace=helloworld frps-ingress
Name:             frps-ingress
Namespace:        helloworld
Address:          192.168.0.176
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                Path  Backends
  ----                ----  --------
  dev.helloworld.com
                      /   frps:vhost (192.168.103.147:8080)
Annotations:          nginx.ingress.kubernetes.io/rewrite-target: /
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    13m (x2 over 13m)  nginx-ingress-controller  Scheduled for sync

But when I ran curl -H 'HOST: dev.helloworld.com' http://localhost:32038/, it return 502:

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

I exexcuted following command in nginx pod, and got correct content:

curl http://frps.helloworld:8080
<!DOCTYPE html>
<html lang="en">
  // ...
</html>

So I'm sure the frps service is correctly running. I have tried not only using header but also change cloud dns record(under another domain), but got same result. What happened, and why ingress-nginx return 502?


I have see something wrong, it seems nginx found wrong upstream. The cluster ip of frps service is 10.97.67.98, while ingress-nginx use pod ip 192.168.103.147. The pod ip is unreachable while service cluster ip is reachable. I don't known why and how to fix it.

-- Vista Chyi
kubernetes
nginx-ingress

1 Answer

12/9/2020

I didn't think that this problem may be caused by the firewall daemon of the host. After enbale masquerade and open ports of kubernetes, everything work fine :) Something like:

firewall-cmd --add-masquerade --permanent
firewall-cmd --permanent --zone=public --add-port=10250-10252/tcp
firewall-cmd --permanent --zone=public --add-port=10255/tcp
firewall-cmd --permanent --zone=public --add-port=30000-32767/tcp
firewall-cmd --reload
-- Vista Chyi
Source: StackOverflow