I'm trying to start a kubelet on a fedora 24/lxc container, but getting an error which appears to be related to libvirt/iptables
Docker (installed using dnf/yum):
[root@node2 ~]# docker version
Client:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built:
OS/Arch: linux/amd64
Server:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built:
OS/Arch: linux/amd64
Kubernetes (downloaded v1.3.3 and extracted tar):
root@node2 bin]# ./kubectl version
Client Version: version.Info{
Major:"1", Minor:"3", GitVersion:"v1.3.3",
GitCommit:"c6411395e09da356c608896d3d9725acab821418",
GitTreeState:"clean", BuildDate:"2016-07-22T20:29:38Z",
GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Startup, params, and error:
[root@node2 bin]# ./kubelet --address=0.0.0.0 --api-servers=http://master1:8080 --container-runtime=docker --hostname-override=node1 --port=10250
I0802 17:43:04.264454 2348 docker.go:327] Start docker client with request timeout=2m0s
W0802 17:43:04.271850 2348 server.go:487] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.
W0802 17:43:04.271906 2348 server.go:448] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.
I0802 17:43:04.272241 2348 manager.go:138] cAdvisor running in container: "/"
W0802 17:43:04.275956 2348 manager.go:146] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused
I0802 17:43:04.280283 2348 fs.go:139] Filesystem partitions: map[/dev/mapper/fedora_kg--fedora-root:{mountpoint:/ major:253 minor:0 fsType:ext4 blockSize:0}]
I0802 17:43:04.284868 2348 manager.go:192] Machine: {NumCores:4 CpuFrequency:3192789
MemoryCapacity:4125679616 MachineID:1e80444278b7442385a762b9545cec7b
SystemUUID:5EC24D56-9CA6-B237-EE21-E0899C3C16AB BootID:44212209-ff1d-4340-8433-11a93274d927
Filesystems:[{Device:/dev/mapper/fedora_kg--fedora-root
Capacity:52710469632 Type:vfs Inodes:3276800}]
DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:85899345920 Scheduler:cfq}
253:0:{Name:dm-0 Major:253 Minor:0 Size:53687091200 Scheduler:none}
253:1:{Name:dm-1 Major:253 Minor:1 Size:4160749568 Scheduler:none}
253:2:{Name:dm-2 Major:253 Minor:2 Size:27518828544 Scheduler:none}
253:3:{Name:dm-3 Major:253 Minor:3 Size:107374182400 Scheduler:none}]
NetworkDevices:[
{Name:eth0 MacAddress:00:16:3e:b9:ce:f3 Speed:10000 Mtu:1500}
{Name:flannel.1 MacAddress:fa:ed:34:75:d6:1d Speed:0 Mtu:1450}]
Topology:[
{Id:0 Memory:4125679616
Cores:[{Id:0 Threads:[0]
Caches:[]} {Id:1 Threads:[1] Caches:[]}]
Caches:[{Size:8388608 Type:Unified Level:3}]}
{Id:1 Memory:0 Cores:[{Id:0 Threads:[2]
Caches:[]} {Id:1 Threads:[3] Caches:[]}]
Caches:[{Size:8388608 Type:Unified Level:3}]}]
CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I0802 17:43:04.285649 2348 manager.go:198]
Version: {KernelVersion:4.6.4-301.fc24.x86_64 ContainerOsVersion:Fedora 24 (Twenty Four)
DockerVersion:1.12.0 CadvisorVersion: CadvisorRevision:}
I0802 17:43:04.286366 2348 server.go:768] Watching apiserver
W0802 17:43:04.286477 2348 kubelet.go:561] Hairpin mode set to "promiscuous-bridge" but configureCBR0 is false, falling back to "hairpin-veth"
I0802 17:43:04.286575 2348 kubelet.go:384] Hairpin mode set to "hairpin-veth"
W0802 17:43:04.303188 2348 plugins.go:170] can't set sysctl net/bridge/bridge-nf-call-iptables: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory
I0802 17:43:04.307700 2348 docker_manager.go:235] Setting dockerRoot to /var/lib/docker
I0802 17:43:04.310175 2348 server.go:730] Started kubelet v1.3.3
E0802 17:43:04.311636 2348 kubelet.go:933] Image garbage collection failed: unable to find data for container /
E0802 17:43:04.312800 2348 kubelet.go:994] Failed to start ContainerManager [open /proc/sys/kernel/panic: read-only file system, open /proc/sys/kernel/panic_on_oops: read-only file system, open /proc/sys/vm/overcommit_memory: read-only file system]
I0802 17:43:04.312962 2348 status_manager.go:123] Starting to sync pod status with apiserver
I0802 17:43:04.313080 2348 kubelet.go:2468] Starting kubelet main sync loop.
I0802 17:43:04.313187 2348 kubelet.go:2477] skipping pod synchronization - [Failed to start ContainerManager [open /proc/sys/kernel/panic: read-only file system, open /proc/sys/kernel/panic_on_oops: read-only file system, open /proc/sys/vm/overcommit_memory: read-only file system] network state unknown container runtime is down]
I0802 17:43:04.313525 2348 server.go:117] Starting to listen on 0.0.0.0:10250
I0802 17:43:04.315021 2348 volume_manager.go:216] Starting Kubelet Volume Manager
I0802 17:43:04.325998 2348 factory.go:228] Registering Docker factory
E0802 17:43:04.326049 2348 manager.go:240] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused
I0802 17:43:04.326073 2348 factory.go:54] Registering systemd factory
I0802 17:43:04.326545 2348 factory.go:86] Registering Raw factory
I0802 17:43:04.326993 2348 manager.go:1072] Started watching for new ooms in manager
I0802 17:43:04.331164 2348 oomparser.go:185] oomparser using systemd
I0802 17:43:04.331904 2348 manager.go:281] Starting recovery of all containers
I0802 17:43:04.368958 2348 manager.go:286] Recovery completed
I0802 17:43:04.419959 2348 kubelet.go:1185] Node node1 was previously registered
I0802 17:43:09.313871 2348 kubelet.go:2477] skipping pod synchronization - [Failed to start ContainerManager [open /proc/sys/kernel/panic: read-only file system, open /proc/sys/kernel/panic_on_oops: read-only file system, open /proc/sys/vm/overcommit_memory: read-only file system]]
Flannel (installed using dnf/yum):
root@node2 bin]# systemctl status flanneld
● flanneld.service - Flanneld overlay address etcd agent
Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 22:14:06 UTC; 21h ago
Process: 1203 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)
Main PID: 1195 (flanneld)
Tasks: 11 (limit: 512)
Memory: 2.7M
CPU: 4.012s
CGroup: /system.slice/flanneld.service
└─1195 /usr/bin/flanneld -etcd-endpoints=http://master1:2379 -etcd-prefix=/flannel/network
LXC settings for the container:
[root@kg-fedora node2]# cat config
# Template used to create this container: /usr/share/lxc/templates/lxc-fedora
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
lxc.network.type = veth
lxc.network.link = virbr0
lxc.network.hwaddr = 00:16:3e:b9:ce:f3
lxc.network.flags = up
lxc.network.ipv4 = 192.168.122.23/24
lxc.network.ipv4.gateway = 192.168.80.2
# Include common configuration
lxc.include = /usr/share/lxc/config/fedora.common.conf
lxc.arch = x86_64
# When using LXC with apparmor, uncomment the next line to run unconfined:
#lxc.aa_profile = unconfined
# example simple networking setup, uncomment to enable
#lxc.network.type = veth
#lxc.network.flags = up
#lxc.network.link = lxcbr0
#lxc.network.name = eth0
# Additional example for veth network type
# static MAC address,
#lxc.network.hwaddr = 00:16:3e:77:52:20
# persistent veth device name on host side
# Note: This may potentially collide with other containers of same name!
#lxc.network.veth.pair = v-fedora-template-e0
lxc.cgroup.devices.allow = a
lxc.cap.drop =
lxc.rootfs = /var/lib/lxc/node2/rootfs
lxc.rootfs.backend = dir
lxc.utsname = node2
libvirt-1.3.3.2-1.fc24.x86_64:
[root@kg-fedora node2]# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2016-07-29 16:33:09 EDT; 3 days ago
Docs: man:libvirtd(8)
http://libvirt.org
Main PID: 1191 (libvirtd)
Tasks: 18 (limit: 512)
Memory: 7.3M
CPU: 9.108s
CGroup: /system.slice/libvirtd.service
├─1191 /usr/sbin/libvirtd
├─1597 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
└─1599 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
Flannel/Docker config:
[root@node2 ~]# systemctl stop docker
[root@node2 ~]# ip link delete docker0
[root@node2 ~]# systemctl start docker
[root@node2 ~]# ip -4 a|grep inet
inet 127.0.0.1/8 scope host lo
inet 10.100.72.0/16 scope global flannel.1
inet 172.17.0.1/16 scope global docker0
inet 192.168.122.23/24 brd 192.168.122.255 scope global dynamic eth0
Notice that the docker0 interface is not using the same ip range as the flannel.1 interface
Any pointers would be much appreciated!
For anyone who may look for the solution to this issue: Since you are using LXC, you need to make sure that the filesystem in question is mounted as rw. This it is needed to specify the following option in the config file for LXC:
raw.lxc: "lxc.apparmor.profile=unconfined\nlxc.cap.drop= \nlxc.cgroup.devices.allow=a\nlxc.mount.auto=proc:rw sys:rw"
or just
lxc.mount.auto: proc:rw sys:rw
Here are the references: https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f04aa94b6c9c https://github.com/corneliusweig/kubernetes-lxd