I'm trying to run a stateful Mongo inside Kubernetes and it works with these configurations outside of Istio.
mongodb: {
uri: "mongodb://mongo-0.mongo,mongo-1.mongo,mongo-2.mongo",
dbName: "app"
}
But when I run the node application inside Istio, it loses the ability to connect to the mongo. Is there something I'm missing or is it so that I can't use stateful-sets with Istio yet?
stateful mongo config below.
apiVersion: v1
kind: Service
metadata:
labels:
service: mongo
name: mongo
spec:
ports:
- name: tcp
port: 27017
targetPort: 27017
clusterIP: None
selector:
service: mongo
role: mongo
***********
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: "mongo"
replicas: 3
template:
metadata:
labels:
role: mongo
environment: test
service: mongo
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongo
image: mongo:3.4.6
resources:
requests:
cpu: "10m"
command:
- mongod
- "--replSet"
- rs0
- "--smallfiles"
- "--noprealloc"
ports:
- containerPort: 27017
volumeMounts:
- name: mongo-persistent-storage
mountPath: /data/db
- name: mongo-sidecar
image: cvallance/mongo-k8s-sidecar
resources:
requests:
cpu: "10m"
env:
- name: MONGO_SIDECAR_POD_LABELS
value: "role=mongo,environment=test"
Error that I'm receiving is
[2017-07-25 12:01:11] ERROR Mongoose MonboDB connection error: {
"message": "write EPIPE",
"name": "MongoError",
"stack": "Error: write EPIPE\n at exports._errnoException (util.js:1024:11)\n at WriteWrap.afterWrite [as oncomplete] (net.js:851:14)"
}
[2017-07-25 12:01:11] ERROR ... retrying createConnection in 5 seconds...
[2017-07-25 12:01:16] ERROR Mongoose MonboDB connection error: {
"message": "read ECONNRESET",
"name": "MongoError",
"stack": "Error: read ECONNRESET\n at exports._errnoException (util.js:1024:11)\n at TCP.onread (net.js:610:25)"
}
StatefulSet support has been recently added, but it's not in a release yet.
For ones who are receiving such error in istio with manual injection:
When you are injecting proxy to your deployment, network is dropped while istio-proxy is configured:
kubectl apply -f kubernetes-configs/
istioctl kube-inject -f kubernetes-configs/deployment.yml | kubectl -n foo apply -f -
In that time, you will receive:
Error: write EPIPE
I solved problem by adding pause before initialization like in this question: How Can I Wait In Node.js (Javascript), l need to pause for a period of time
UPDATE
Here is the another way. Just let nodejs app restart. By invoking process.exit(137)
in any place where connection is not successful, Docker container will be killed and restarted. In that time, istio-proxy will finish network configuration and your app will successfully connect to DB.