K
Q

kubectl error while connecting to Rancher Kube API server behind nginx

5/27/2021

My setup has Rancher and Nginx. Nginx handles SSL Certificate and forwards requests to Rancher.

Rancher recommends using this config in Nginx:

enter image description here

The nginx config is done as recommended by rancher. It looks like this:

location / {

    proxy_pass http://localhost:8082/;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    # This allows the ability for the execute shell window to remain open for up to 15 minutes. Without this parameter, the default is 1 minute and will automatically close.
    proxy_read_timeout 900s;
    proxy_buffering off;
}

However I get this error when I use kubectl:

$ kubectl get all
Error from server (InternalError): an error on the server ("invalid upgrade response: status code 200") has prevented the request from succeeding

Since the error message points to some problem related to "upgrade", I comment these two lines from the nginx config above and reload nginx:

#proxy_set_header Upgrade $http_upgrade;
#proxy_set_header Connection "upgrade";

After this, kubectl starts working properly but Rancher UI gives this error:

Error connecting to WebSocket Unable to establish a WebSocket connection to the server. If your server is behind a proxy or SSL-termination device, Browser can not connect to WebSocket. If you run the server behind a proxy, please make sure the proxy supports WebSockets. Streaming stats, logs, shell/console, and auto-updating of the state of resources may not work until this is resolved.

How do I make both - kubectl and Rancher UI - work properly?

-- Dojo
kubectl
kubernetes
nginx
rancher

Similar Questions

How access variable declared earlier in same cloudbuild.yaml file
PDO establishing connection to Azure PostgreSQL is very slow
Kubernetes: find Java application specific logs in one pod when application crashes
Should all pods using a redis cache be constrained to the same node as the rediscache itself?
Kubernetes ExternalName service not visible in DNS

1 Answer

5/27/2021

I solved this by creating another nginx location configuration for URLs accessed by kubectl, and commented out other customizations.

Rancher hosts the Kube API at https://mydomain/k8s/clusters/mycluster

Assuming URLs beginning with /k8s/ don't require Websockets or anything fancy, I disabled upgrade (and subsequently all customizations) for this section of the URL space.

The working config looks like this:

location / {
    proxy_pass http://localhost:8082/;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    # This allows the ability for the execute shell window to remain open for up to 15 minutes. Without this parameter, the default is 1 minute and will automatically close.
    proxy_read_timeout 900s;
    proxy_buffering off;
}

location /k8s/ {
    proxy_pass http://localhost:8082/k8s/;
}
-- Dojo
Source: StackOverflow