K
Q

calico network connectivity failing between pods and services and pods in different hosts

11/7/2020

I have implemented a multi master HA kubernetes cluster and wanted to implement the Calico the hardway as described in here. I was able complete all the steps and my connectivity is not there between the pods and services and pods and other pods in different nodes.

only, difference is I use two different AZs in AWS and I suppose it should not be an issue. I can see pods are getting the IPs and calico network interfaces are creating but still see the connectivity as I explained. Node even doesn't have the public internet access. I did the BGP configuration exactly same in the guide but no luck and I'm not quite sure something to be changed in the BGP configuration when it comes to multi-AZ deployment. I'm not much aware of the Calico BGP configuration.

Unfortunately, calicoctl node diags does not properly run and not providing much more information to move forward.

I'd love here your valuable thoughts and constructive criticism to fix this.

-- Aruna Lakmal
bgp
calico
cni
kubernetes
project-calico

Similar Questions

Ingress rule does not work with Service of type LoadBalancer
Kubernetes replace deployment fails
azure-kubernetes-service fails at creation on webconsole
Nginx request for two or more nodes in Kubernetes
Kubectl connectivity issue

1 Answer

11/10/2020

Calico configured in BGP mode requires all of the instances to be located in the same subnet to work out of the box.

To use calico with deployments that are split across multiple availability zones you must:

Disable AWS source / destination check (see here):

You can do that using AWS CLI: 

	aws ec2 modify-instance-attribute --no-source-dest-check --instance-id 			$EC2_INSTANCE_ID --region <REGION-WHERE-EC2-INSTANCE-IS-LAUNCHED>

Or using the AWS console:

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
  2. In the navigation pane, choose Instances.
  3. Select the NAT instance, choose Actions, Networking, Change Source/Dest. Check.
  4. For the NAT instance, verify that this attribute is disabled. Otherwise, choose Yes, Disable.
  5. If the NAT instance has a secondary network interface, choose it from Network interfaces on the Description tab and choose the interface ID to go to the network interfaces page. Choose Actions, Change Source/Dest. Check, disable the setting, and choose Save.*

Enable IPIP encapsulation and outgoing NAT on your Calico IP pools

(IPPool) represents a collection of IP addresses from which Calico expects endpoint IPs to be assigned. (see here how to set it up)

, then all of the Kubernetes instances must be located in the same subnet for Calico to work out of the box.

To enable the “CrossSubnet” IPIP feature, configure your Calico IP pool resources to enable IPIP mode to “CrossSubnet” like in the example below: 

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: ippool-multi-az
spec:
  cidr: 192.168.0.0/16
  ipipMode: CrossSubnet
EOF

Example above refers to AWS cloud configuration taken from the Calico documentation. Please note that Calico docs has also information about GCP, Azure and IBM.

Remark: If you face another problems going "the hard way" you may want to use as a reference another cluster created by following calico guides below:

Lastly, it is worth to check is also this very good document about calico routing modes (it shows also cross subnets ipip mode).

-- acid_fuji
Source: StackOverflow