I am new to Google Container Engine (GKE). When run on localhost it's working fine but when I deploy to production with GKE I got websocket error.
My node app is develop with Hapi.js and Socket.io and my structure is shown in image below.
I'm using Glue to compose Hapi server. Below is my manifest.json
{
...
"connections": [
{
"host": "app",
"address": "0.0.0.0",
"port": 8000,
"labels": ["api"],
"routes": {
"cors": false,
"security": {
"hsts": false,
"xframe": true,
"xss": true,
"noOpen": true,
"noSniff": true
}
},
"router": {
"stripTrailingSlash": true
},
"load": {
"maxHeapUsedBytes": 1073741824,
"maxRssBytes": 1610612736,
"maxEventLoopDelay": 5000
}
},
{
"host": "app",
"address": "0.0.0.0",
"port": 8099,
"labels": ["web"],
"routes": {
"cors": true,
"security": {
"hsts": false,
"xframe": true,
"xss": true,
"noOpen": true,
"noSniff": true
}
},
"router": {
"stripTrailingSlash": true
},
"load": {
"maxHeapUsedBytes": 1073741824,
"maxRssBytes": 1610612736,
"maxEventLoopDelay": 5000
}
},
{
"host": "app",
"address": "0.0.0.0",
"port": 8999,
"labels": ["admin"],
"routes": {
"cors": true,
"security": {
"hsts": false,
"xframe": true,
"xss": true,
"noOpen": true,
"noSniff": true
}
},
"router": {
"stripTrailingSlash": true
},
"load": {
"maxHeapUsedBytes": 1073741824,
"maxRssBytes": 1610612736,
"maxEventLoopDelay": 5000
},
"state": {
"ttl": null,
"isSecure": false,
"isHttpOnly": true,
"path": null,
"domain": null,
"encoding": "none",
"clearInvalid": false,
"strictHeader": true
}
}
],
...
}
And my nginx.conf
worker_processes 5; ## Default: 1
worker_rlimit_nofile 8192;
error_log /dev/stdout info;
events {
worker_connections 4096; ## Default: 1024
}
http {
access_log /dev/stdout;
server {
listen 80 default_server;
listen [::]:80 default_server;
# Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.
return 301 https://$host$request_uri;
}
server {
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
server_name _;
# Configure ssl
ssl_certificate /etc/secret/ssl/myapp.com.csr;
ssl_certificate_key /etc/secret/ssl/myapp.com.key;
include /etc/nginx/ssl-params.conf;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name api.myapp.com;
location / {
proxy_pass http://api_app/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Handle Web Socket connections
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name myapp.com;
location / {
proxy_pass http://web_app/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Handle Web Socket connections
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name admin.myapp.com;
location / {
proxy_pass http://admin_app/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Handle Web Socket connections
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
# Define your "upstream" servers - the
# servers request will be sent to
upstream api_app {
server localhost:8000;
}
upstream web_app {
server localhost:8099;
}
upstream admin_app {
server localhost:8999;
}
}
Kubernetes service app-service.yaml
apiVersion: v1
kind: Service
metadata:
name: app-nginx
labels:
app: app-nginx
spec:
type: LoadBalancer
ports:
# The port that this service should serve on.
- port: 80
targetPort: 80
protocol: TCP
name: http
- port: 443
targetPort: 443
protocol: TCP
name: https
# Label keys and values that must match in order to receive traffic for this service.
selector:
app: app-nginx
Kubernetes Deployment app-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: app-nginx
spec:
replicas: 3
template:
metadata:
labels:
app: app-nginx
spec:
containers:
- name: nginx
image: us.gcr.io/myproject/nginx
ports:
- containerPort: 80
name: http
- containerPort: 443
name: https
volumeMounts:
# This name must match the volumes.name below.
- name: ssl-secret
readOnly: true
mountPath: /etc/secret/ssl
- name: app
image: us.gcr.io/myproject/bts-server
ports:
- containerPort: 8000
name: api
- containerPort: 8099
name: web
- containerPort: 8999
name: admin
volumeMounts:
# This name must match the volumes.name below.
- name: client-secret
readOnly: true
mountPath: /etc/secret/client
- name: admin-secret
readOnly: true
mountPath: /etc/secret/admin
volumes:
- name: ssl-secret
secret:
secretName: ssl-key-secret
- name: client-secret
secret:
secretName: client-key-secret
- name: admin-secret
secret:
secretName: admin-key-secretAnd I'm using Cloudflare SSL full strict.
Error get from Browser console:
WebSocket connection to 'wss://api.myapp.com/socket.io/?EIO=3&transport=websocket&sid=4Ky-y9K7J0XotrBFAAAQ' failed: WebSocket is closed before the connection is established.
https://api.myapp.com/socket.io/?EIO=3&transport=polling&t=LYByND2&sid=4Ky-y9K7J0XotrBFAAAQ Failed to load resource: the server responded with a status of 400 ()
VM50:35 WebSocket connection to 'wss://api.myapp.com/socket.io/?EIO=3&transport=websocket&sid=FsCGx-UE7ohrsSSqAAAT' failed: Error during WebSocket handshake: Unexpected response code: 502WrappedWebSocket @ VM50:35WS.doOpen @ socket.io.js:6605Transport.open @ socket.io.js:4695Socket.probe @ socket.io.js:3465Socket.onOpen @ socket.io.js:3486Socket.onHandshake @ socket.io.js:3546Socket.onPacket @ socket.io.js:3508(anonymous function) @ socket.io.js:3341Emitter.emit @ socket.io.js:6102Transport.onPacket @ socket.io.js:4760callback @ socket.io.js:4510(anonymous function) @ socket.io.js:5385exports.decodePayloadAsBinary @ socket.io.js:5384exports.decodePayload @ socket.io.js:5152Polling.onData @ socket.io.js:4514(anonymous function) @ socket.io.js:4070Emitter.emit @ socket.io.js:6102Request.onData @ socket.io.js:4231Request.onLoad @ socket.io.js:4312xhr.onreadystatechange @ socket.io.js:4184
socket.io.js:4196 GET https://api.myapp.com/socket.io/?EIO=3&transport=polling&t=LYByNpy&sid=FsCGx-UE7ohrsSSqAAAT 400 ()And here is Nginx's logs:
[22/Nov/2016:12:10:19 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=MGc--oncQbQI6NOZAAAX HTTP/1.1" 101 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
10.8.0.1 - - [22/Nov/2016:12:10:19 +0000] "POST /socket.io/?EIO=3&transport=polling&t=LYByQBw&sid=MGc--oncQbQI6NOZAAAX HTTP/1.1" 200 2 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
10.128.0.2 - - [22/Nov/2016:12:10:20 +0000] "GET /socket.io/?EIO=3&transport=polling&t=LYByQKp HTTP/1.1" 200 101 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
10.8.0.1 - - [22/Nov/2016:12:10:21 +0000] "GET /socket.io/?EIO=3&transport=polling&t=LYByQWo&sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1" 200 24 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
2016/11/22 12:10:21 [error] 6#6: *157 connect() failed (111: Connection refused) while connecting to upstream, client: 10.8.0.1, server: api.myapp.com, request: "GET /socket.io/?EIO=3&transport=polling&t=LYByQaN&sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1", upstream: "http://[::1]:8000/socket.io/?EIO=3&transport=polling&t=LYByQaN&sid=c5nkusT9fEPRsu2rAAAY", host: "api.myapp.com", referrer: "https://myapp.com/"
2016/11/22 12:10:21 [warn] 6#6: *157 upstream server temporarily disabled while connecting to upstream, client: 10.8.0.1, server: api.myapp.com, request: "GET /socket.io/?EIO=3&transport=polling&t=LYByQaN&sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1", upstream: "http://[::1]:8000/socket.io/?EIO=3&transport=polling&t=LYByQaN&sid=c5nkusT9fEPRsu2rAAAY", host: "api.myapp.com", referrer: "https://myapp.com/"
10.8.0.1 - - [22/Nov/2016:12:10:22 +0000] "GET /socket.io/?EIO=3&transport=polling&t=LYByQaN&sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1" 200 4 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"UPDATE
When I change replicas to 1 in app-deployment.yaml it's work. But I think it's not a good solution. I need 3 replicas.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: app-nginx
spec:
replicas: 1
template:
metadata:
labels:
app: app-nginxHow to make it work with 3 replicas?
After I update Kubernetes service template to use sessionAffinity: ClientIP it works now. But just get some error when first press Ctrl + F5 and on second press it's work fine.
Error during WebSocket handshake: Unexpected response code: 400However, I still get data from server. So I think it's okay.
Updated Service template
apiVersion: v1
kind: Service
metadata:
name: app-nginx
labels:
app: app-nginx
spec:
sessionAffinity: ClientIP
type: LoadBalancer
ports:
# The port that this service should serve on.
- port: 80
targetPort: 80
protocol: TCP
name: http
- port: 443
targetPort: 443
protocol: TCP
name: https
# Label keys and values that must match in order
# to receive traffic for this service.
selector:
app: app-nginx