I have a Java pod application which connect to CosmosDB using the default connection string. The pod runs perfectly on my local minikube. But it has connection exception when I deploy it to my AKS.
The pod keep crashing with this MongoDB error.
Timed out after 30000 ms while waiting for server that matches... Client view of cluster state is ... xxx.documents.azure.com:10255
Looks like it cannot reach the CosmosDB. At first I think it is because the default Network security rules block the outgoing port 10255. Then I add a NSG to that resource group. Add the outgoing rule on port 10255. It does not solve the problem.
Then I stumble upon this article. CosmosDB on AKS using OSBA Is it the only way? do I have to use OSBA to access a public CosmosDB?
Connection string copy from Azure portal
mongodb://mycompany:some_base64_encrypted_stuff@mycompany.documents.azure.com:10255/?ssl=true&replicaSet=globaldb
UPDATE:
turns out the spring-boot-starter 2.1.0 is using mongodb-driver 3.8.2. the mongodb-driver appends :27017 to my connection string. I have updated it to 3.10.2. Now that the connection string is correct. My program running in Kubernetes is giving me UnknownHostException mydoc.documents.azure.com. I am guessing there could be a problem caused by build docker image in windows and then run it on alpine.
UPDATE:
I think I am getting very close to the answer. The problem is from the kubernetes cluster. My cluster contains two nodes. I create another cluster with single node. Deploy my program to it and it connects to CosmosDB without error. But I do not know how to debug kubernetes cluster.
if you are using network policies, you can use the sample network policy to allow all egress traffic:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all
spec:
podSelector: {}
egress:
- {}
policyTypes:
- Egress
If you are not using Network policies, connection should just work, you dont need OSBA. You can fine tune the network policy later to make it more restrictive.
https://kubernetes.io/docs/concepts/services-networking/network-policies/
ps. if you are using istio with certain settings your outbound requests will be also blocker, you'd need to account for that as well