I have a containerised application & I am trying to pass the header proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; from my nginx to my rails application.

but when i use this header, the readiness and liveliness probes start failing with status 502 and awaiting headers and my deployment does not successfully complete.

Here is the outoput of kubectl describe pod where we see 502

  Warning  Unhealthy  11m                   kubelet, gke-cluster-1-upgraded-pool--v82p  Liveness probe failed: Get http://10.16.0.1:3000/users/sign_in: dial tcp 10.16.0.221:3000: connect: connection refused
  Warning  Unhealthy  11m                   kubelet, gke-cluster-1-upgraded-pool--v82p  Readiness probe failed: HTTP probe failed with statuscode: 502
  Warning  Unhealthy  11m                   kubelet, gke-cluster-1-upgraded-pool--v82p  Liveness probe failed: HTTP probe failed with statuscode: 502
  Warning  Unhealthy  11m                   kubelet, gke-cluster-1-upgraded-pool--v82p  Readiness probe failed: Get http://10.16.0.1:3000/users/sign_in: dial tcp 10.16.0.221:3000: connect: connection refused
  Warning  Unhealthy  11m                   kubelet, gke-cluster-1-upgraded-pool--v82p  Liveness probe failed: Get http://10.16.0.1:3000/users/sign_in: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  BackOff    84s (x33 over 8m56s)  kubelet, gke-cluster-1-upgraded-pool--v82p  Back-off restarting failed container

Previously the header value was proxy_set_header X-Forwarded-For $http_x_forwarded_for;

below is the relevant part of my nginx.conf

http {
  set_real_ip_from 1.2.3.4; -- example ip
  real_ip_header X-Forwarded-For;
  real_ip_recursive on;

  server {
    listen 80 default_server;
    server_name _;
    client_max_body_size 16m;
    location @app {
      proxy_pass http://127.0.0.1:3000;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
    }
  }
}

below is my deployment file, i have tried to keep only the relevant parts and removed volumne mounts and db connection settings

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: backend

spec:
  template:
    metadata:
      labels:
        app: backend
    spec:
      initContainers:
        - name: nginx-config
          image: my_nginx
          command: ['/bin/sh', '-c']
          args: ["sed -i -e 's/gzip_types/gzip_types application\\/json/g' /etc/nginx/nginx.conf && mv /etc/nginx/* /etc/nginx-new/"]
          volumeMounts:
            - name: nginx-config
              mountPath: /etc/nginx-new
        - name: copy-assets
          image: unicorn_image
          command: ['sh', '-c', 'cp -a /app/public/* /mnt/']
          volumeMounts:
            - name: assets
              mountPath: /mnt
      containers:
        - name: unicorn
          image: unicorn_image
          ports:
            - containerPort: 3000
          env:
          livenessProbe:
            httpGet:
              path: /users/sign_in
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10  
          readinessProbe:
            httpGet:
              path: /users/sign_in
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10  
        - name: nginx
          image: my_nginx
          ports:
            - containerPort: 80
          livenessProbe:
            httpGet:
              path: /users/sign_in
              port: 80
              scheme: HTTP
            initialDelaySeconds: 10  
          readinessProbe:
            httpGet:
              path: /users/sign_in
              port: 80
              scheme: HTTP
            initialDelaySeconds: 10  
          volumeMounts:
            - name: assets
              mountPath: /var/www
              readOnly: true
            - name: nginx-config
              mountPath: /etc/nginx

Update: i see that the nginx container is not getting created. Please see below

kubectl exec -it backend-6f4974cfd6-jtnqk -c nginx -- /bin/bash error: unable to upgrade connection: container not found ("nginx")

Not sure as to how updating a header is causing an issue.

can anyone please point me in the right direction as to how i can fix it, Thanks.