Ansible: Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory

1/6/2019

I'm setting up kubernetes cluster with ansible. I get the following error when trying to enable kernel IP routing:

Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory

Is this a bug in ansible or is there something wrong with my playbook?

---
# file: site.yml
# description: Asentaa ja kaynnistaa kubernetes-klusterin riippuvuuksineen
#
# resources:
#   - https://kubernetes.io/docs/setup/independent/install-kubeadm/
#   - http://michele.sciabarra.com/2018/02/12/devops/Kubernetes-with-KubeAdm-Ansible-Vagrant/
#   - https://docs.ansible.com/ansible/latest/modules/
#   - https://github.com/geerlingguy/ansible-role-kubernetes/blob/master/tasks/setup-RedHat.yml
#   - https://docs.docker.com/install/linux/docker-ce/centos/
#
# author: Tuomas Toivonen
# date: 30.12.2018

- name: Asenna docker ja kubernetes

  hosts: k8s-machines
  become: true
  become_method: sudo

  roles:
    - common

  vars:
    ip_modules:
      - ip_vs
      - ip_vs_rr
      - ip_vs_wrr
      - ip_vs_sh
      - nf_conntrack_ipv4

  tasks:
    - name: Poista swapfile
      tags:
        - os-settings
      mount:
        name: swap
        fstype: swap
        state: absent

    - name: Disabloi swap-muisti
      tags:
        - os-settings
      command: swapoff -a
      when: ansible_swaptotal_mb > 0

    - name: Konfiguroi verkkoasetukset
      tags:
        - os-settings
      command: modprobe {{ item }}
      loop: "{{ ip_modules }}"

    - name: Modprobe
      tags:
        - os-settings
      lineinfile:
        path: "/etc/modules"
        line: "{{ item }}"
        create: yes
        state: present
      loop: "{{ ip_modules }}"

    - name: Iptables
      tags:
        - os-settings
      sysctl:
          name: "{{ item }}"
          value: 1
          sysctl_set: yes
          state: present
          reload: yes
      loop:
        - 'net.bridge.bridge-nf-call-iptables'
        - 'net.bridge.bridge-nf-call-ip6tables'

    - name: Salli IP-reititys
      sysctl:
        name: net.ipv4.ip_forward
        value: 1
        state: present
        reload: yes
        sysctl_set: yes


    - name: Lisaa docker-ce -repositorio
      tags:
        - repos
      yum_repository:
        name: docker-ce
        description: docker-ce
        baseurl: https://download.docker.com/linux/centos/7/x86_64/stable/
        enabled: true
        gpgcheck: true
        repo_gpgcheck: true
        gpgkey:
          - https://download.docker.com/linux/centos/gpg
        state: present

    - name: Lisaa kubernetes -repositorio
      tags:
        - repos
      yum_repository:
        name: kubernetes
        description: kubernetes
        baseurl: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
        enabled: true
        gpgcheck: true
        repo_gpgcheck: true
        gpgkey:
          - https://packages.cloud.google.com/yum/doc/yum-key.gpg
          - https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
        state: present

    - name: Asenna docker-ce -paketti
      tags:
        - packages
      yum:
        name: docker-ce
        state: present

    - name: Asenna NTP -paketti
      tags:
        - packages
      yum:
        name: ntp
        state: present

    - name: Asenna kubernetes -paketit
      tags:
        - packages
      yum:
        name: "{{ item }}"
        state: present
      loop:
        - kubelet
        - kubeadm
        - kubectl

    - name: Kaynnista palvelut
      tags:
        - services
      service: name={{ item }} state=started enabled=yes
      loop:
        - docker
        - ntpd
        - kubelet

- name: Alusta kubernetes masterit
  become: true
  become_method: sudo
  hosts: k8s-masters
  tags:
    - cluster

  tasks:
    - name: kubeadm reset
      shell: "kubeadm reset -f"

    - name: kubeadm init
      shell: "kubeadm init --token-ttl=0 --apiserver-advertise-address=10.0.0.101 --pod-network-cidr=20.0.0.0/8" # TODO
      register: kubeadm_out

    - set_fact:
        kubeadm_join: "{{ kubeadm_out.stdout_lines[-1] }}"
      when: kubeadm_out.stdout.find("kubeadm join") != -1

    - debug:
        var: kubeadm_join

    - name: Aseta ymparistomuuttujat
      shell: >
          cp /etc/kubernetes/admin.conf /home/vagrant/ &&
          chown vagrant:vagrant /home/vagrant/admin.conf &&
          export KUBECONFIG=/home/vagrant/admin.conf &&
          echo export KUBECONFIG=$KUBECONFIG >> /home/vagrant/.bashrc

- name: Konfiguroi CNI-verkko
  become: true
  become_method: sudo
  hosts: k8s-masters
  tags:
    - cluster-network

  tasks:
    - sysctl: name=net.bridge.bridge-nf-call-iptables value=1 state=present reload=yes sysctl_set=yes
    - sysctl: name=net.bridge.bridge-nf-call-ip6tables value=1 state=present reload=yes sysctl_set=yes
    - name: Asenna Flannel-plugin
      shell: >
        export KUBECONFIG=/home/vagrant/admin.conf ;
        kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

    - shell: sleep 10

- name: Alusta kubernetes workerit
  become: true
  become_method: sudo
  hosts: k8s-workers
  tags:
    - cluster

  tasks:
    - name: kubeadm reset
      shell: "kubeadm reset -f"

    - name: kubeadm join
      tags:
        - cluster
      shell: "{{ hostvars['k8s-n1'].kubeadm_join }}" # TODO

Here is the full ansible log

ansible-controller: Running ansible-playbook...
cd /vagrant && PYTHONUNBUFFERED=1 ANSIBLE_NOCOLOR=true ANSIBLE_CONFIG='ansible/ansible.cfg' ansible-playbook --limit="all" --inventory-file=ansible/hosts -v ansible/site.yml
Using /vagrant/ansible/ansible.cfg as config file
/vagrant/ansible/hosts did not meet host_list requirements, check plugin documentation if this is unexpected
/vagrant/ansible/hosts did not meet script requirements, check plugin documentation if this is unexpected

PLAY [Asenna docker ja kubernetes] *********************************************

TASK [Gathering Facts] *********************************************************
ok: [k8s-n1]
ok: [k8s-n3]
ok: [k8s-n2]

TASK [common : Testaa] *********************************************************
changed: [k8s-n3] => {"changed": true, "checksum": "6920e1826e439962050ec0ab4221719b3a045f04", "dest": "/template.test", "gid": 0, "group": "root", "md5sum": "a4f61c365318c3e23d466914fbd02687", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 14, "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1546760756.54-124542112178019/source", "state": "file", "uid": 0}
changed: [k8s-n2] => {"changed": true, "checksum": "6920e1826e439962050ec0ab4221719b3a045f04", "dest": "/template.test", "gid": 0, "group": "root", "md5sum": "a4f61c365318c3e23d466914fbd02687", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 14, "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1546760756.51-240329169302936/source", "state": "file", "uid": 0}
changed: [k8s-n1] => {"changed": true, "checksum": "6920e1826e439962050ec0ab4221719b3a045f04", "dest": "/template.test", "gid": 0, "group": "root", "md5sum": "a4f61c365318c3e23d466914fbd02687", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 14, "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1546760756.57-121244542660821/source", "state": "file", "uid": 0}

TASK [common : Asenna telnet] **************************************************
changed: [k8s-n2] => {"changed": true, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: ftp.funet.fi\n * extras: ftp.funet.fi\n * updates: ftp.funet.fi\nResolving Dependencies\n--> Running transaction check\n---> Package telnet.x86_64 1:0.17-64.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package          Arch             Version                 Repository      Size\n================================================================================\nInstalling:\n telnet           x86_64           1:0.17-64.el7           base            64 k\n\nTransaction Summary\n================================================================================\nInstall  1 Package\n\nTotal download size: 64 k\nInstalled size: 113 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n  Installing : 1:telnet-0.17-64.el7.x86_64                                  1/1 \n  Verifying  : 1:telnet-0.17-64.el7.x86_64                                  1/1 \n\nInstalled:\n  telnet.x86_64 1:0.17-64.el7                                                   \n\nComplete!\n"]}
changed: [k8s-n1] => {"changed": true, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: centos.mirror.gnu.fi\n * extras: centos.mirror.gnu.fi\n * updates: centos.mirror.gnu.fi\nResolving Dependencies\n--> Running transaction check\n---> Package telnet.x86_64 1:0.17-64.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package          Arch             Version                 Repository      Size\n================================================================================\nInstalling:\n telnet           x86_64           1:0.17-64.el7           base            64 k\n\nTransaction Summary\n================================================================================\nInstall  1 Package\n\nTotal download size: 64 k\nInstalled size: 113 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n  Installing : 1:telnet-0.17-64.el7.x86_64                                  1/1 \n  Verifying  : 1:telnet-0.17-64.el7.x86_64                                  1/1 \n\nInstalled:\n  telnet.x86_64 1:0.17-64.el7                                                   \n\nComplete!\n"]}
changed: [k8s-n3] => {"changed": true, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: ftp.funet.fi\n * extras: ftp.funet.fi\n * updates: ftp.funet.fi\nResolving Dependencies\n--> Running transaction check\n---> Package telnet.x86_64 1:0.17-64.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package          Arch             Version                 Repository      Size\n================================================================================\nInstalling:\n telnet           x86_64           1:0.17-64.el7           base            64 k\n\nTransaction Summary\n================================================================================\nInstall  1 Package\n\nTotal download size: 64 k\nInstalled size: 113 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n  Installing : 1:telnet-0.17-64.el7.x86_64                                  1/1 \n  Verifying  : 1:telnet-0.17-64.el7.x86_64                                  1/1 \n\nInstalled:\n  telnet.x86_64 1:0.17-64.el7                                                   \n\nComplete!\n"]}

TASK [Poista swapfile] *********************************************************
ok: [k8s-n1] => {"changed": false, "dump": "0", "fstab": "/etc/fstab", "fstype": "swap", "name": "swap", "opts": "defaults", "passno": "0"}
ok: [k8s-n2] => {"changed": false, "dump": "0", "fstab": "/etc/fstab", "fstype": "swap", "name": "swap", "opts": "defaults", "passno": "0"}
ok: [k8s-n3] => {"changed": false, "dump": "0", "fstab": "/etc/fstab", "fstype": "swap", "name": "swap", "opts": "defaults", "passno": "0"}

TASK [Disabloi swap-muisti] ****************************************************
changed: [k8s-n3] => {"changed": true, "cmd": ["swapoff", "-a"], "delta": "0:00:00.009581", "end": "2019-01-06 07:46:08.414842", "rc": 0, "start": "2019-01-06 07:46:08.405261", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => {"changed": true, "cmd": ["swapoff", "-a"], "delta": "0:00:00.119638", "end": "2019-01-06 07:46:08.484265", "rc": 0, "start": "2019-01-06 07:46:08.364627", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n2] => {"changed": true, "cmd": ["swapoff", "-a"], "delta": "0:00:00.133924", "end": "2019-01-06 07:46:08.519646", "rc": 0, "start": "2019-01-06 07:46:08.385722", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [Konfiguroi verkkoasetukset] **********************************************
changed: [k8s-n2] => (item=ip_vs) => {"changed": true, "cmd": ["modprobe", "ip_vs"], "delta": "0:00:00.036881", "end": "2019-01-06 07:46:10.606797", "item": "ip_vs", "rc": 0, "start": "2019-01-06 07:46:10.569916", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n3] => (item=ip_vs) => {"changed": true, "cmd": ["modprobe", "ip_vs"], "delta": "0:00:00.036141", "end": "2019-01-06 07:46:10.815043", "item": "ip_vs", "rc": 0, "start": "2019-01-06 07:46:10.778902", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => (item=ip_vs) => {"changed": true, "cmd": ["modprobe", "ip_vs"], "delta": "0:00:00.035888", "end": "2019-01-06 07:46:10.768267", "item": "ip_vs", "rc": 0, "start": "2019-01-06 07:46:10.732379", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n2] => (item=ip_vs_rr) => {"changed": true, "cmd": ["modprobe", "ip_vs_rr"], "delta": "0:00:00.005942", "end": "2019-01-06 07:46:12.763004", "item": "ip_vs_rr", "rc": 0, "start": "2019-01-06 07:46:12.757062", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => (item=ip_vs_rr) => {"changed": true, "cmd": ["modprobe", "ip_vs_rr"], "delta": "0:00:00.006084", "end": "2019-01-06 07:46:12.896763", "item": "ip_vs_rr", "rc": 0, "start": "2019-01-06 07:46:12.890679", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n3] => (item=ip_vs_rr) => {"changed": true, "cmd": ["modprobe", "ip_vs_rr"], "delta": "0:00:00.006325", "end": "2019-01-06 07:46:12.899750", "item": "ip_vs_rr", "rc": 0, "start": "2019-01-06 07:46:12.893425", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n2] => (item=ip_vs_wrr) => {"changed": true, "cmd": ["modprobe", "ip_vs_wrr"], "delta": "0:00:00.006195", "end": "2019-01-06 07:46:14.795507", "item": "ip_vs_wrr", "rc": 0, "start": "2019-01-06 07:46:14.789312", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => (item=ip_vs_wrr) => {"changed": true, "cmd": ["modprobe", "ip_vs_wrr"], "delta": "0:00:00.007328", "end": "2019-01-06 07:46:14.819072", "item": "ip_vs_wrr", "rc": 0, "start": "2019-01-06 07:46:14.811744", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n3] => (item=ip_vs_wrr) => {"changed": true, "cmd": ["modprobe", "ip_vs_wrr"], "delta": "0:00:00.007251", "end": "2019-01-06 07:46:14.863192", "item": "ip_vs_wrr", "rc": 0, "start": "2019-01-06 07:46:14.855941", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n3] => (item=ip_vs_sh) => {"changed": true, "cmd": ["modprobe", "ip_vs_sh"], "delta": "0:00:00.007590", "end": "2019-01-06 07:46:16.815226", "item": "ip_vs_sh", "rc": 0, "start": "2019-01-06 07:46:16.807636", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => (item=ip_vs_sh) => {"changed": true, "cmd": ["modprobe", "ip_vs_sh"], "delta": "0:00:00.006380", "end": "2019-01-06 07:46:16.941470", "item": "ip_vs_sh", "rc": 0, "start": "2019-01-06 07:46:16.935090", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n2] => (item=ip_vs_sh) => {"changed": true, "cmd": ["modprobe", "ip_vs_sh"], "delta": "0:00:00.006619", "end": "2019-01-06 07:46:16.808432", "item": "ip_vs_sh", "rc": 0, "start": "2019-01-06 07:46:16.801813", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n3] => (item=nf_conntrack_ipv4) => {"changed": true, "cmd": ["modprobe", "nf_conntrack_ipv4"], "delta": "0:00:00.007618", "end": "2019-01-06 07:46:18.825593", "item": "nf_conntrack_ipv4", "rc": 0, "start": "2019-01-06 07:46:18.817975", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => (item=nf_conntrack_ipv4) => {"changed": true, "cmd": ["modprobe", "nf_conntrack_ipv4"], "delta": "0:00:00.008181", "end": "2019-01-06 07:46:18.910050", "item": "nf_conntrack_ipv4", "rc": 0, "start": "2019-01-06 07:46:18.901869", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n2] => (item=nf_conntrack_ipv4) => {"changed": true, "cmd": ["modprobe", "nf_conntrack_ipv4"], "delta": "0:00:00.007427", "end": "2019-01-06 07:46:18.962850", "item": "nf_conntrack_ipv4", "rc": 0, "start": "2019-01-06 07:46:18.955423", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [Modprobe] ****************************************************************
changed: [k8s-n2] => (item=ip_vs) => {"backup": "", "changed": true, "item": "ip_vs", "msg": "line added"}
changed: [k8s-n1] => (item=ip_vs) => {"backup": "", "changed": true, "item": "ip_vs", "msg": "line added"}
changed: [k8s-n3] => (item=ip_vs) => {"backup": "", "changed": true, "item": "ip_vs", "msg": "line added"}
changed: [k8s-n2] => (item=ip_vs_rr) => {"backup": "", "changed": true, "item": "ip_vs_rr", "msg": "line added"}
changed: [k8s-n1] => (item=ip_vs_rr) => {"backup": "", "changed": true, "item": "ip_vs_rr", "msg": "line added"}
changed: [k8s-n3] => (item=ip_vs_rr) => {"backup": "", "changed": true, "item": "ip_vs_rr", "msg": "line added"}
changed: [k8s-n2] => (item=ip_vs_wrr) => {"backup": "", "changed": true, "item": "ip_vs_wrr", "msg": "line added"}
changed: [k8s-n1] => (item=ip_vs_wrr) => {"backup": "", "changed": true, "item": "ip_vs_wrr", "msg": "line added"}
changed: [k8s-n3] => (item=ip_vs_wrr) => {"backup": "", "changed": true, "item": "ip_vs_wrr", "msg": "line added"}
changed: [k8s-n2] => (item=ip_vs_sh) => {"backup": "", "changed": true, "item": "ip_vs_sh", "msg": "line added"}
changed: [k8s-n1] => (item=ip_vs_sh) => {"backup": "", "changed": true, "item": "ip_vs_sh", "msg": "line added"}
changed: [k8s-n3] => (item=ip_vs_sh) => {"backup": "", "changed": true, "item": "ip_vs_sh", "msg": "line added"}
changed: [k8s-n2] => (item=nf_conntrack_ipv4) => {"backup": "", "changed": true, "item": "nf_conntrack_ipv4", "msg": "line added"}
changed: [k8s-n1] => (item=nf_conntrack_ipv4) => {"backup": "", "changed": true, "item": "nf_conntrack_ipv4", "msg": "line added"}
changed: [k8s-n3] => (item=nf_conntrack_ipv4) => {"backup": "", "changed": true, "item": "nf_conntrack_ipv4", "msg": "line added"}

TASK [Iptables] ****************************************************************
failed: [k8s-n3] (item=net.bridge.bridge-nf-call-iptables) => {"changed": false, "item": "net.bridge.bridge-nf-call-iptables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\n"}
failed: [k8s-n1] (item=net.bridge.bridge-nf-call-iptables) => {"changed": false, "item": "net.bridge.bridge-nf-call-iptables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\n"}
failed: [k8s-n2] (item=net.bridge.bridge-nf-call-iptables) => {"changed": false, "item": "net.bridge.bridge-nf-call-iptables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\n"}
failed: [k8s-n3] (item=net.bridge.bridge-nf-call-ip6tables) => {"changed": false, "item": "net.bridge.bridge-nf-call-ip6tables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\nsysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory\n"}
failed: [k8s-n2] (item=net.bridge.bridge-nf-call-ip6tables) => {"changed": false, "item": "net.bridge.bridge-nf-call-ip6tables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\nsysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory\n"}
failed: [k8s-n1] (item=net.bridge.bridge-nf-call-ip6tables) => {"changed": false, "item": "net.bridge.bridge-nf-call-ip6tables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\nsysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory\n"}
        to retry, use: --limit @/vagrant/ansible/site.retry

PLAY RECAP *********************************************************************
k8s-n1                     : ok=7    changed=5    unreachable=0    failed=1
k8s-n2                     : ok=7    changed=5    unreachable=0    failed=1
k8s-n3                     : ok=7    changed=5    unreachable=0    failed=1

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
-- Tuomas Toivonen
ansible
iptables
kernel
kubernetes
sysctl

1 Answer

1/6/2019

Loading br_netfilter kernel module fixed the problem. I simply appended it to the ip_modules list in playbook vars declaration. I'm using Centos 7.

-- Tuomas Toivonen
Source: StackOverflow