I setup a k8s cluster on my server(all in one node) and installed ingress-nginx. Now I want to deploy a frps service to the cluster. Here is my code:
apiVersion: v1
kind: Namespace
metadata:
name: helloworld
---
apiVersion: v1
kind: ConfigMap
metadata:
name: frps-configmap
namespace: helloworld
data:
frps.ini: |
[common]
bind_port = 7000
vhost_http_port = 8080
custom_404_page = /etc/frp/frps.html
frps.html: |
<!DOCTYPE html>
<html lang="en">
// ...
</html>
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: frps-deployment
namespace: helloworld
labels:
app: frps
spec:
replicas: 1
selector:
matchLabels:
app: frps
template:
metadata:
labels:
app: frps
spec:
containers:
- name: frps
image: 'snowdreamtech/frps:0.34.3'
ports:
- containerPort: 8080
- containerPort: 7000
volumeMounts:
- name: frps-etc
mountPath: '/etc/frp'
readOnly: true
volumes:
- name: frps-etc
configMap:
name: frps-configmap
items:
- key: frps.ini
path: frps.ini
- key: frps.html
path: frps.html
---
apiVersion: v1
kind: Service
metadata:
name: frps
namespace: helloworld
spec:
selector:
app: frps
type: ClusterIP
ports:
- protocol: TCP
port: 8080
targetPort: 8080
name: vhost
- protocol: TCP
port: 7000
targetPort: 7000
name: frps
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: frps-ingress
namespace: helloworld
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: 'nginx'
rules:
- host: dev.helloworld.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frps
port:
name: vhostAnd here are some cluster information after apply previous yaml:
kubectl get deploy -A
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
helloworld frps-deployment 1/1 1 1 12m
ingress-nginx ingress-nginx-controller 1/1 1 1 23d
...
kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
helloworld frps ClusterIP 10.97.67.98 <none> 8080/TCP,7000/TCP 6m18s
ingress-nginx ingress-nginx-controller NodePort 10.101.128.200 <none> 80:32038/TCP,443:31363/TCP 23d
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.97.36.119 <none> 443/TCP 23d
...
kubectl describe ingress --namespace=helloworld frps-ingress
Name: frps-ingress
Namespace: helloworld
Address: 192.168.0.176
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
dev.helloworld.com
/ frps:vhost (192.168.103.147:8080)
Annotations: nginx.ingress.kubernetes.io/rewrite-target: /
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 13m (x2 over 13m) nginx-ingress-controller Scheduled for syncBut when I ran curl -H 'HOST: dev.helloworld.com' http://localhost:32038/, it return 502:
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>I exexcuted following command in nginx pod, and got correct content:
curl http://frps.helloworld:8080
<!DOCTYPE html>
<html lang="en">
// ...
</html>So I'm sure the frps service is correctly running. I have tried not only using header but also change cloud dns record(under another domain), but got same result. What happened, and why ingress-nginx return 502?
I have see something wrong, it seems nginx found wrong upstream. The cluster ip of frps service is 10.97.67.98, while ingress-nginx use pod ip 192.168.103.147. The pod ip is unreachable while service cluster ip is reachable. I don't known why and how to fix it.
I didn't think that this problem may be caused by the firewall daemon of the host. After enbale masquerade and open ports of kubernetes, everything work fine :) Something like:
firewall-cmd --add-masquerade --permanent
firewall-cmd --permanent --zone=public --add-port=10250-10252/tcp
firewall-cmd --permanent --zone=public --add-port=10255/tcp
firewall-cmd --permanent --zone=public --add-port=30000-32767/tcp
firewall-cmd --reload