we want to use netdata to monitor application performance, but w/o update each application image, so want to use postStart hook to achieve this.
we can use postStart to echo some logs, but using poststart can not startup netdata successfully with following config:
image: 10.18.210.178:40080/k8s-deploy/netdata:test4
imagePullPolicy: IfNotPresent
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- sleep 10; /usr/sbin/netdata -p 19999 -u ssdepg
but netdata can startup w/o the sleep 10 commands:
image: 10.18.210.178:40080/k8s-deploy/netdata:test4
imagePullPolicy: IfNotPresent
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- /usr/sbin/netdata -p 19999 -u ssdepg
with both config, the application POD can go into RUNING state, only difference we can not see netdata process with 1st case.
there is no any error indication for pod describe, netdata logs or k8s logs.
can any expert give us some hint why sleep cause this.
need clarify that there is no any issue for command line usage, see netdata logs as below, it is started by k8s via postStart, but did not succeed ( can not find process via "ps" command).
2019-07-18 08:44:59: netdata INFO : MAIN : Executing /usr/libexec/netdata/plugins.d/system-info.sh
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_OS_NAME="CentOS Linux"
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_OS_ID=centos
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_OS_ID_LIKE=rhel fedora
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_OS_VERSION=7 (Core)
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_OS_VERSION_ID=7
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_OS_DETECTION=/etc/os-release
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=3.10.0-327.el7.x86_64
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=none
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=systemd-detect-virt
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER=none
2019-07-18 08:44:59: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=systemd-detect-virt
2019-07-18 08:44:59: netdata INFO : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'START' '-' '-'
2019-07-18 08:45:01: netdata ERROR : MAIN : child pid 56 exited with code 28.
2019-07-18 08:45:01: netdata INFO : MAIN : resources control: allowed file descriptors: soft = 655360, max = 655360
2019-07-18 08:45:01: netdata INFO : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value 999
2019-07-18 08:45:01: netdata INFO : MAIN : Adjusted netdata scheduling policy to idle (5), with priority 0.
2019-07-18 08:45:01: netdata INFO : MAIN : Running with process scheduling policy 'idle'
2019-07-18 08:45:01: netdata INFO : MAIN : netdata started on pid 83.
2019-07-18 08:45:01: netdata INFO : MAIN : CONFIG: cannot load user config '/etc/netdata/stream.conf'. Will try stock config.
2019-07-18 08:45:01: netdata INFO : MAIN : Host 'nginx-test-0717-1003812089-288d5' (at registry as 'nginx-test-0717-1003812089-288d5') with guid '54cb87fe-a938-11e9-8cc8-ca282c4f3765' initialized, os 'linux',
timezone 'UTC', tags '', program_name 'netdata', program_version 'v1.15.0', update every 5, memory mode save, history entries 924, streaming disabled (to '' with api key ''), health disabled, cache_dir '/var/ca
che/netdata', varlib_dir '/var/lib/netdata', health_log '/var/lib/netdata/health/health-log.db', alarms default handler '/usr/libexec/netdata/plugins.d/alarm-notify.sh', alarms default recipient 'root'
2019-07-18 08:45:01: netdata INFO : MAIN : SYSTEM_INFO: free 0x1057e90
2019-07-18 08:45:01: netdata INFO : PLUGIN[proc] : thread created with task id 84
2019-07-18 08:45:01: netdata INFO : STATSD : thread created with task id 85
2019-07-18 08:45:01: netdata INFO : BACKENDS : thread created with task id 86
2019-07-18 08:45:01: netdata INFO : WEB_SERVER[static1] : thread created with task id 87
2019-07-18 08:45:01: netdata INFO : MAIN : netdata initialization completed. Enjoy real-time performance monitoring!
2019-07-18 08:45:01: netdata INFO : HEALTH : thread created with task id 89
2019-07-18 08:45:01: netdata INFO : PLUGINSD : thread created with task id 88
2019-07-18 08:45:01: netdata INFO : PLUGINSD[apps] : thread created with task id 90
2019-07-18 08:45:01: netdata ERROR : PLUGINSD : cannot open plugins directory '/etc/netdata/custom-plugins.d' (errno 2, No such file or directory)
2019-07-18 08:45:01: netdata INFO : PLUGINSD[apps] : connected to '/usr/libexec/netdata/plugins.d/apps.plugin' running on pid 91
2019-07-18 08:45:01: netdata INFO : WEB_SERVER[static1] : 2019-07-18 08:45:01: apps.plugin ERROR : MAIN : PROCFILE: Cannot open file '/etc/netdata/apps_groups.conf' (errno 2, No such file or directory)
2019-07-18 08:45:01: apps.plugin INFO : MAIN : Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'
2019-07-18 08:45:01: apps.plugin INFO : MAIN : Loaded config file '/usr/lib/netdata/conf.d/apps_groups.conf'
2019-07-18 08:45:01: apps.plugin INFO : MAIN : started on pid 91
It worth to make it as suggested in official K8s documentation
image: 10.18.210.178:40080/k8s-deploy/netdata:test4
imagePullPolicy: IfNotPresent
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "sleep 10; /usr/sbin/netdata -p 19999 -u ssdepg"
This can work
command:
- bash
- "-c"
- |
sleep 10
/usr/sbin/netdata -p 19999 -u ssdepg