google/cadvisor:latest image monitor container run error

10/6/2016

I found the bug below on the above "devicemapper" at bugzilla.redhat.com. It let me install the "device-mapper-persistent-data" package. However, when I installed it, I received the following error.

I tried to use this link to solve the error, however, I still haven't managed to solve it.

This is what I tried:

1. update the libcontainer dependency
2. look at all cgroup mounts

Here is the error:

[root@slave712 ~]# docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
google/cadvisor:latest
Unable to find image 'google/cadvisor:latest' locally
latest: Pulling from google/cadvisor
e110a4a17941: Pull complete 
e17fa94aae07: Pull complete 
7097ce2382c0: Pull complete 
Digest:     sha256:3b1f3714a27d40bf413d5fde0429ad6410bc14fcdae5a3478daa34cc701901fa
Status: Downloaded newer image for google/cadvisor:latest
a54c09247fd6572f96365dced4178f0cec8696329ed39c0349991f39ab6d660d

[root@slave712 ~]# docker logs cadvisor

I1001 02:41:16.237284 1 storagedriver.go:50] Caching stats in memory for 2m0s
I1001 02:41:16.238324 1 manager.go:140] cAdvisor running in container: "/docker/a54c09247fd6572f96365dced4178f0cec8696329ed39c0349991f39ab6d660d"
W1001 02:41:16.379106 1 manager.go:148] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I1001 02:41:16.428790 1 fs.go:116] Filesystem partitions: map[/dev/mapper/centos-root:{mountpoint:/var/lib/docker/devicemapper major:253 minor:0 fsType:xfs blockSize:0} /dev/sda1:{mountpoint:/rootfs/boot major:8 minor:1 fsType:xfs blockSize:0} /dev/mapper/docker-253:0-402876333-34a64b9159b501f40b949fc1ae894ee7fc1985645d65a63f4dfe689c32919e43:{mountpoint:/ major:253 minor:3 fsType:xfs blockSize:0}]
I1001 02:41:16.655301 1 manager.go:195] Machine: {NumCores:4 CpuFrequency:3192628 MemoryCapacity:1025396736 MachineID:f40694ca9e71417fb1af85dcf66dfc39 SystemUUID:564DBE01-1C91-8E79-5C04-6FC1D390B5EB BootID:9008dc53-91fe-4510-b199-8c26241bc0b5 Filesystems:[{Device:/dev/mapper/docker-253:0-402876333-34a64b9159b501f40b949fc1ae894ee7fc1985645d65a63f4dfe689c32919e43 Capacity:10725883904 Type:vfs Inodes:10484736 HasInodes:true} {Device:/dev/mapper/centos-root Capacity:105011822592 Type:vfs Inodes:102600704 HasInodes:true} {Device:/dev/sda1 Capacity:206221312 Type:vfs Inodes:204800 HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size:105063120896 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:2097152000 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:107374182400 Scheduler:none} 253:3:{Name:dm-3 Major:253 Minor:3 Size:10737418240 Scheduler:none} 2:0:{Name:fd0 Major:2 Minor:0 Size:4096 Scheduler:deadline} 8:0:{Name:sda Major:8 Minor:0 Size:107374182400 Scheduler:deadline}] NetworkDevices:[{Name:br-92607f6f2739 MacAddress:02:42:9a:50:1f:2b Speed:0 Mtu:1500} {Name:eno16777736 MacAddress:00:0c:29:90:b5:eb Speed:1000 Mtu:1500}] Topology:[{Id:0 Memory:1073205248 Cores:[{Id:0 Threads:[0] Caches:[]} {Id:1 Threads:[1] Caches:[]}] Caches:[{Size:6291456 Type:Unified Level:3}]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[2] Caches:[]} {Id:1 Threads:[3] Caches:[]}] Caches:[{Size:6291456 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I1001 02:41:16.659951 1 manager.go:201] Version: {KernelVersion:3.10.0-327.el7.x86_64 ContainerOsVersion:Alpine Linux v3.4 DockerVersion:1.12.1 CadvisorVersion:v0.24.0 CadvisorRevision:0cdf491}
E1001 02:41:16.698803 1 factory.go:291] devicemapper filesystem stats will not be reported: unable to find thin_ls binary
I1001 02:41:16.698900 1 factory.go:295] Registering Docker factory
W1001 02:41:16.698962 1 manager.go:244] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I1001 02:41:16.698982 1 factory.go:54] Registering systemd factory
I1001 02:41:16.699503 1 factory.go:86] Registering Raw factory
I1001 02:41:16.700006 1 manager.go:1082] Started watching for new ooms in manager
W1001 02:41:16.715521 1 manager.go:272] Could not configure a source for OOM detection, disabling OOM events: unable to find any kernel log file available from our set: [/var/log/kern.log /var/log/messages /var/log/syslog]
I1001 02:41:16.730767 1 manager.go:285] Starting recovery of all containers
I1001 02:41:19.183769 1 manager.go:290] Recovery completed
F1001 02:41:19.254307 1 cadvisor.go:151] Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

[root@slave712 ~]# docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built:

OS/Arch: linux/amd64

Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built:

OS/Arch: linux/amd64

[root@slave712 ~]# mount -l
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=490460k,nr_inodes=122615,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/centos-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
/dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=100140k,mode=700)
/dev/mapper/centos-root on /var/lib/docker/devicemapper type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

[root@slave712 ~]# uname -a
Linux slave712 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@slave712 ~]# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core)
-- Jason Tom
cadvisor
cgroups
device-mapper
docker
kubernetes

1 Answer

10/6/2016

It looks like you're hitting the issue fixed in the link you mentioned. That fix has not yet been incorporated into a cAdvisor release, so it is not included when you run the docker image google/cadvisor:latest (assuming you didn't build your own image and tag it with that).

You can try building and running your own docker image from master, or wait for the next patch release (should be soon), which will include that fix.

EDIT:

The fix has now been incorporated into cAdvisor v0.24.1 (docker image google/cadvisor:v0.24.1)

-- Tim Allclair
Source: StackOverflow