K
Q

Why are there many exited containers in Kubernetes after some times?

6/22/2021

Sometimes when I run docker ps -a, I see about 5-10 containers in Exited status. For example, after one or two hours I see them.

Why are there such these exited containers and why are they in Exited status?

9f6bd4fa5a05   8522d622299c             "/opt/bin/flanneld -…"   About a minute ago   Exited (1) About a minute ago             k8s_kube-flannel_kube-flannel-ds-x8s8v_kube-system_024c58c6-9cc2-4c2a-b0d3-12a49101a57b_3
7203f2beaaef   8522d622299c             "cp -f /etc/kube-fla…"   2 minutes ago        Exited (0) About a minute ago             k8s_install-cni_kube-flannel-ds-x8s8v_kube-system_024c58c6-9cc2-4c2a-b0d3-12a49101a57b_1
e4ade2e2617c   09708983cc37             "kube-controller-man…"   2 minutes ago        Exited (2) 2 minutes ago                  k8s_kube-controller-manager_kube-controller-manager-localhost_kube-system_00f8734ea34e7389b55cb740d1fcf000_2
1a3762bfa98c   k8s.gcr.io/pause:3.4.1   "/pause"                 2 minutes ago        Exited (0) 2 minutes ago                  k8s_POD_kube-controller-manager-localhost_kube-system_00f8734ea34e7389b55cb740d1fcf000_2
65555c2db5c6   nginx                    "/docker-entrypoint.…"   5 hours ago          Exited (0) 2 minutes ago                  k8s_task-pv-container_task-pv-pod_default_8bfa648a-ddb0-4531-8984-81d4ce8db444_0
a6b99bcc020a   k8s.gcr.io/pause:3.4.1   "/pause"                 5 hours ago          Exited (0) 2 minutes ago                  k8s_POD_task-pv-pod_default_8bfa648a-ddb0-4531-8984-81d4ce8db444_0
b625c68f4fce   296a6d5035e2             "/coredns -conf /etc…"   6 hours ago          Exited (0) 2 minutes ago                  k8s_coredns_coredns-558bd4d5db-c88d8_kube-system_6170d444-f992-4163-aff0-480cd1ed1ce4_1
7897a49ca8c6   296a6d5035e2             "/coredns -conf /etc…"   6 hours ago          Exited (0) 2 minutes ago                  k8s_coredns_coredns-558bd4d5db-z8kfs_kube-system_b34f29d6-bd72-40ba-a4ba-102e71965f98_1
9b7cac87b4bb   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_coredns-558bd4d5db-z8kfs_kube-system_b34f29d6-bd72-40ba-a4ba-102e71965f98_10
02661c9b507e   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_coredns-558bd4d5db-c88d8_kube-system_6170d444-f992-4163-aff0-480cd1ed1ce4_10
0f31cf13fc3a   38ddd85fe90e             "/usr/local/bin/kube…"   6 hours ago          Exited (2) 2 minutes ago                  k8s_kube-proxy_kube-proxy-s9rf6_kube-system_ff2a1ef4-7316-47c4-8584-aac1a1cd63d5_1
91a53859a423   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_kube-proxy-s9rf6_kube-system_ff2a1ef4-7316-47c4-8584-aac1a1cd63d5_1
a4833a1301a4   4d217480042e             "kube-apiserver --ad…"   6 hours ago          Exited (137) 2 minutes ago                k8s_kube-apiserver_kube-apiserver-localhost_kube-system_bbc60c89c6c6215aca670678072d707c_1
e4d76060736a   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_kube-apiserver-localhost_kube-system_bbc60c89c6c6215aca670678072d707c_1
7c7f09ab3633   0369cf4303ff             "etcd --advertise-cl…"   6 hours ago          Exited (0) 2 minutes ago                  k8s_etcd_etcd-localhost_kube-system_146e48c8e9d3b464d1f47c594a8becc8_1
0204740e75b0   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_etcd-localhost_kube-system_146e48c8e9d3b464d1f47c594a8becc8_1
8631dd16c767   62ad3129eca8             "kube-scheduler --au…"   6 hours ago          Exited (2) 2 minutes ago                  k8s_kube-scheduler_kube-scheduler-localhost_kube-system_7d7118d190aeaaf065b4e86b4982f05f_1
a59ee59475cd   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_kube-scheduler-localhost_kube-system_7d7118d190aeaaf065b4e86b4982f05f_1

The log of one of them:

docker logs k8s_kube-proxy_kube-proxy-s9rf6_kube-system_ff2a1ef4-7316-47c4-8584-aac1a1cd63d5_1
I0622 09:48:20.971080       1 node.go:172] Successfully retrieved node IP: 192.168.43.85
I0622 09:48:20.971170       1 server_others.go:140] Detected node IP 192.168.43.85
W0622 09:48:20.971200       1 server_others.go:592] Unknown proxy mode "", assuming iptables proxy
I0622 09:48:25.066916       1 server_others.go:206] kube-proxy running in dual-stack mode, IPv4-primary
I0622 09:48:25.067112       1 server_others.go:212] Using iptables Proxier.
I0622 09:48:25.067154       1 server_others.go:219] creating dualStackProxier for iptables.
W0622 09:48:25.067185       1 server_others.go:506] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
I0622 09:48:25.187175       1 server.go:643] Version: v1.21.0
I0622 09:48:25.407303       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0622 09:48:25.407351       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0622 09:48:25.407727       1 conntrack.go:83] Setting conntrack hashsize to 32768
I0622 09:48:25.477081       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0622 09:48:25.477170       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0622 09:48:25.517411       1 config.go:315] Starting service config controller
I0622 09:48:25.517679       1 config.go:224] Starting endpoint slice config controller
I0622 09:48:25.531370       1 shared_informer.go:240] Waiting for caches to sync for service config
I0622 09:48:25.548740       1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
W0622 09:48:25.665718       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 09:48:25.667197       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
I0622 09:48:25.731527       1 shared_informer.go:247] Caches are synced for service config 
I0622 09:48:25.753275       1 shared_informer.go:247] Caches are synced for endpoint slice config 
W0622 09:54:21.682298       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:01:18.747213       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:10:27.734119       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:18:50.736315       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:28:33.903205       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:36:05.938074       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:43:23.940703       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:49:20.951579       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:58:54.953921       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 11:08:28.961630       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 11:14:26.988498       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 11:20:12.010192       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 11:29:09.023270       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:00:28.838837       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:05:46.840726       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:10:50.843578       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:19:07.846779       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:26:46.848443       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:34:57.853658       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:42:00.887206       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:50:03.935106       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:57:18.937677       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 15:03:41.955870       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 15:11:26.957845       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 15:17:51.979285       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
docker logs k8s_kube-flannel_kube-flannel-ds-x8s8v_kube-system_024c58c6-9cc2-4c2a-b0d3-12a49101a57b_3
I0622 15:19:21.070951       1 main.go:520] Determining IP address of default interface
I0622 15:19:21.071786       1 main.go:533] Using interface with name enp0s3 and address 192.168.43.85
I0622 15:19:21.073223       1 main.go:550] Defaulting external address to interface address (192.168.43.85)
W0622 15:19:21.073336       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E0622 15:19:31.157536       1 main.go:251] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-x8s8v': Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-x8s8v": net/http: TLS handshake timeout
-- Saeed
docker
kubernetes

Similar Questions

Console logs for the nginx ui service in Openshift written in Zulu time stamp - want them in EST
How can I mount a dynamically generated file into a pod without creating a ConfigMap?
How do I setup Ingress controller for Kubernetes where one end point for test environment and another for production environment?
getting CrashLoopBackOff Error for 1/4 pods due to Error syncing pod
AWS Image for Kubernetes

1 Answer

6/23/2021

You are using deprecated API:

discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice

Based on this documentation:

EndpointSlice

The discovery.k8s.io/v1beta1 API version of EndpointSlice will no longer be served in v1.25.

  • Migrate manifests and API clients to use the discovery.k8s.io/v1 API version, available since v1.21.
  • All existing persisted objects are accessible via the new API
  • Notable changes in discovery.k8s.io/v1:
    • use per Endpoint nodeName field instead of deprecated topology["kubernetes.io/hostname"] field
    • use per Endpoint zone field instead of deprecated topology["topology.kubernetes.io/zone"] field
    • topology is replaced with the deprecatedTopology field which is not writable in v1

You need to find all deprecated API and migrate to supported versions.

-- Mikołaj Głodziak
Source: StackOverflow