K
Q

Calico based pod is not coming up

7/12/2020

I am setting up a pod using calico but it keeps on failing with some authorization error. By default following is the node cidr of my system:

[root@k8master-1 ~]# kubeadm config view | grep Subnet
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12

I have set up the ippools using the following steps:

https://docs.projectcalico.org/getting-started/kubernetes/flannel/flannel

IP Pool creation

- apiVersion: projectcalico.org/v3
  kind: IPPool
  metadata:
    name: rack-ip-pool 
  spec:
    blockSize: 26
    cidr: 10.244.1.0/24
    ipipMode: Never
    natOutgoing: true
    nodeSelector: all()
    vxlanMode: Never

Ip Pool List

[root@k8master-1 ~]# calicoctl get ippool -o wide
NAME          CIDR            NAT    IPIPMODE   VXLANMODE   DISABLED   SELECTOR   
rack-ip-pool   10.244.1.0/24   true   Never      Never       false      all()

Pod Yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: testcalico
  labels:
    cracklerack: "1"
spec:
  serviceName: testcalico-svc
  selector:
    matchLabels:
      cracklerack: "1"
  template:
    metadata:
      labels:
        cracklerack: "1"
      annotations:
       cni.projectcalico.org/ipv4pools: "[\"rack-ip-pool\"]"
    spec:
      runtimeClassName: kata-containers
      containers:
      - name: testcalico
        image: cracklelinux:7
        ports:
        - containerPort: 80
        command: [/usr/sbin/init]
        securityContext:
          privileged: true
---
apiVersion: v1
kind: Service
metadata:
 name: testcalico-svc
spec:
 clusterIP: None
 selector:
   cracklerack: "1"

When I create a pod, it throws the following error:

Error

 Warning  FailedCreatePodSandBox  112s  kubelet, k8worker-1  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_xxxxx-0_default_45357eab-bf40-4fe7-a470-da42c9668116_0(579e2c258154fcdc2e85df4a1e35264ea9550b0dd1c4384331abc471f552456d): connection is unauthorized: ipamconfigs.crd.projectcalico.org "default" is forbidden: User "system:serviceaccount:kube-system:canal" cannot get resource "ipamconfigs" in API group "crd.projectcalico.org" at the cluster scope
-- drifter
kubernetes
project-calico

Similar Questions

What is relationship betweeen Node and Namespaces in kubernetes?
Setting up a local Kubernetes installation
kubernetes is not able forward port
troubleshoot Google kubernetes load balancer unhealthy nodes
How to safegaurd prometheus data?

2 Answers

7/12/2020

I used following conf file and it worked:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-node
rules:
  # The CNI plugin needs to get pods, nodes, and namespaces.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - endpoints
      - services
    verbs:
      # Used to discover service IPs for advertisement.
      - watch
      - list
      # Used to discover Typhas.
      - get
  # Pod CIDR auto-detection on kubeadm needs access to config maps.
  - apiGroups: [""]
    resources:
      - configmaps
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - nodes/status
    verbs:
      # Needed for clearing NodeNetworkUnavailable flag.
      - patch
      # Calico stores some configuration information in node annotations.
      - update
  # Watch for changes to Kubernetes NetworkPolicies.
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
    verbs:
      - watch
      - list
  # Used by Calico for policy information.
  - apiGroups: [""]
    resources:
      - pods
      - namespaces
      - serviceaccounts
    verbs:
      - list
      - watch
  # The CNI plugin patches pods/status.
  - apiGroups: [""]
    resources:
      - pods/status
    verbs:
      - patch
  # Calico monitors various CRDs for config.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - globalfelixconfigs
      - felixconfigurations
      - bgppeers
      - globalbgpconfigs
      - bgpconfigurations
      - ippools
      - ipamblocks
      - ipamconfigs
      - globalnetworkpolicies
      - globalnetworksets
      - networkpolicies
      - networksets
      - clusterinformations
      - hostendpoints
      - blockaffinities
    verbs:
      - get
      - list
      - watch
  # Calico must create and update some CRDs on startup.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
      - ipamblocks
      - ipamconfigs
      - blockaffinities
      - felixconfigurations
      - clusterinformations
    verbs:
      - create
      - update
  # Calico stores some configuration information on the node.
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
  # These permissions are only required for upgrade from v2.6, and can
  # be removed after upgrade or on fresh installations.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - bgpconfigurations
      - bgppeers
    verbs:
      - create
      - update

Another Block in the same file:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
rules:
  # Nodes are watched to monitor for deletions.
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - watch
      - list
      - get
  # Pods are queried to check for existence.
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
  # IPAM resources are manipulated when nodes are deleted.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
    verbs:
      - list
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
      - ipamblocks
      - ipamhandles
      - ipamconfigs
    verbs:
      - get
      - list
      - create
      - update
      - delete
  # kube-controllers manages hostendpoints.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - hostendpoints
    verbs:
      - get
      - list
      - create
      - update
      - delete
  # Needs access to update clusterinformations.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - clusterinformations
    verbs:
      - get
      - create
      - update
  # KubeControllersConfiguration is where it gets its config
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - kubecontrollersconfigurations
    verbs:
      # read its own config
      - get
      # create a default if none exists
      - create
      # update status
      - update
      # watch for changes
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-kube-controllers
subjects:
- kind: ServiceAccount
  name: calico-kube-controllers
  namespace: kube-system
---
-- drifter
Source: StackOverflow
7/12/2020

Looks like you have an RBAC issue where your pod cannot read the Kubernetes the IPAMConfig CRD.

I looked at the manifests from https://docs.projectcalico.org/manifests/canal.yaml and I see that it's missing ipamconfigs from a couple of the RBAC ClusterRoles. So you can go ahead and try to add them.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
rules:
  # Nodes are watched to monitor for deletions.
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - watch
      - list
      - get
  # Pods are queried to check for existence.
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
  # IPAM resources are manipulated when nodes are deleted.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
    verbs:
      - list
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
      - ipamblocks
      - ipamhandles
      - ipamconfigs 👈 add here
 ...

Then the other ClusterRole:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-node
rules:
...
  # Calico monitors various CRDs for config.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - globalfelixconfigs
      - felixconfigurations
      - bgppeers
      - globalbgpconfigs
      - bgpconfigurations
      - ippools
      - ipamblocks
      - ipamconfigs 👈 add here
      - globalnetworkpolicies
      - globalnetworksets
      - networkpolicies
      - networksets
      - clusterinformations
      - hostendpoints
      - blockaffinities
    verbs:
      - get
      - list
      - watch
  # Calico must create and update some CRDs on startup.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
      - ipamconfigs 👈 just in case
      - felixconfigurations
      - clusterinformations
    verbs:
      - create
      - update
...

Then run:

kubectl apply -f canal.yaml

After applying this, you might need to restart your cluster (needed on my minikube at least).

-- Rico
Source: StackOverflow