I am deploying the nginx based ingress controller on Kubernetes cluster managed by RKE. ( I have also tried the same directly without RKE ).
In both the cases , it tries to use/bind to Ports 80 , and 443 on the host, and it fails because in the pod security policy for all service accounts I am not allowing host ports.
In fact I don't need to access the ingress directly on the hosts, but I want to access the ingress controller as a Service on the NodePort from external LoadBalancer.
Is there way to deploy Nginx ingress controller not to use any hostPort.
Done by disabling hostNetwork , and remove unnecessary privileges and capabilities:
C02W84XMHTD5:Downloads iahmad$ kubectl get deployments -n ingress-nginx -o yaml
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
name: nginx-ingress-controller
namespace: ingress-nginx
resourceVersion: "68427"
selfLink: /apis/extensions/v1beta1/namespaces/ingress-nginx/deployments/nginx-ingress-controller
uid: 0b92b556-12fa-11ea-9d82-08002762a3c5
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
creationTimestamp: null
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
containers:
- args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
name: nginx-ingress-controller
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
resources: {}
securityContext:
runAsUser: 33
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: nginx-ingress-serviceaccount
serviceAccountName: nginx-ingress-serviceaccount
terminationGracePeriodSeconds: 300
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2019-11-29T22:46:59Z
lastUpdateTime: 2019-11-29T22:46:59Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: 2019-11-29T22:46:13Z
lastUpdateTime: 2019-11-29T22:46:59Z
message: ReplicaSet "nginx-ingress-controller-84758fb96c" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
kind: List
metadata:
resourceVersion: ""
selfLink: ""and then creating a nodeport service pointing to the ingress controller ports:
C02W84XMHTD5:Downloads iahmad$ kubectl get svc -n ingress-nginx -o yaml
apiVersion: v1
items:
- apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
name: ingress-nginx
namespace: ingress-nginx
resourceVersion: "68063"
selfLink: /api/v1/namespaces/ingress-nginx/services/ingress-nginx
uid: 7aa425a4-12f9-11ea-9d82-08002762a3c5
spec:
clusterIP: 10.97.110.93
externalTrafficPolicy: Cluster
ports:
- name: http
nodePort: 30864
port: 80
protocol: TCP
targetPort: 80
- name: https
nodePort: 30716
port: 443
protocol: TCP
targetPort: 443
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
kind: List
metadata:
resourceVersion: ""
selfLink: ""
C02W84XMHTD5:Downloads iahmad$
In the documentation about NodePort, you can find that this type can allocate ports from range 30000-32767. However there is a workaround. If you will add a special flag --service-node-port-range with requested range, admission controller allow you to create NodePort with Ports 80 and 443.
You will need to go to /etc/kubernetes/manifests/, edit kube-apiserver.yaml with sudo and add entry - --service-node-port-range=1-32767. After that you need to save it.
Now you will need to create service. To do that you need to edit this yaml and in ports add node port to spec.ports
Before:
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCPAfter:
ports:
- name: http
nodePort: 80
port: 80
protocol: TCP
targetPort: 80
- name: https
nodePort: 443
port: 443
protocol: TCP
targetPort: 443After thoses changes you can edit again kube-apiserver.yaml in /etc/kubernetes/manifests/ and comment it using # in the same line as - --service-node-port-range.
Then you will be able to curl this NodePort address and Node address.
EDIT: After clarification
Ingress can be deployed in two ways. The first one is deploy Nginx as Deamonset which is requiring hostPort inside configuration file. However there is another option, you can deploy Nginx as Deployment.
NodeIP and Known Port: Pods in the DaemonSet can use a hostPort, so that the pods are reachable via the node IPs. Clients know the list of node IPs somehow, and know the port by convention.
However in the bottom of the page you can find:
DaemonSets are similar to Deployments in that they both create Pods, and those Pods have processes which are not expected to terminate (e.g. web servers, storage servers).
Use a Deployment for stateless services, like frontends, where scaling up and down the number of replicas and rolling out updates are more important than controlling exactly which host the Pod runs on. Use a DaemonSet when it is important that a copy of a Pod always run on all or certain hosts, and when it needs to start before other Pods.
You need to deploy Ingress as Deployment and not as Deamonset.
Example of Nginx Deployment can be found here. As Deployment is not requiring hostPort you will be able to create pods without this parameter.