K
Q

Kubectl: Fatal error runtime out of memory

2/1/2022

I logged into the Linux bastion host where kubectl is installed for connecting to the Kubernetes cluster.

On the Bastion host when I run any kubectl command like the one below:

kubectl get pods

I get the error below:

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x192cb47, 0x16)
        /usr/local/go/src/runtime/panic.go:617 +0x72 fp=0x7fff534ae560 sp=0x7fff534ae530 pc=0x42d482
runtime.sysMap(0xc000000000, 0x4000000, 0x2d178b8)
        /usr/local/go/src/runtime/mem_linux.go:170 +0xc7 fp=0x7fff534ae5a0 sp=0x7fff534ae560 pc=0x417d07
runtime.(*mheap).sysAlloc(0x2cfef80, 0x2000, 0x2cfef90, 0x1)
        /usr/local/go/src/runtime/malloc.go:633 +0x1cd fp=0x7fff534ae648 sp=0x7fff534ae5a0 pc=0x40ae0d
runtime.(*mheap).grow(0x2cfef80, 0x1, 0x0)
        /usr/local/go/src/runtime/mheap.go:1222 +0x42 fp=0x7fff534ae6a0 sp=0x7fff534ae648 pc=0x425022
runtime.(*mheap).allocSpanLocked(0x2cfef80, 0x1, 0x2d178c8, 0x0)
        /usr/local/go/src/runtime/mheap.go:1150 +0x37f fp=0x7fff534ae6d8 sp=0x7fff534ae6a0 pc=0x424f0f
runtime.(*mheap).alloc_m(0x2cfef80, 0x1, 0x2a, 0x0)
        /usr/local/go/src/runtime/mheap.go:977 +0xc2 fp=0x7fff534ae728 sp=0x7fff534ae6d8 pc=0x424562
runtime.(*mheap).alloc.func1()
        /usr/local/go/src/runtime/mheap.go:1048 +0x4c fp=0x7fff534ae760 sp=0x7fff534ae728 pc=0x456a4c
runtime.(*mheap).alloc(0x2cfef80, 0x1, 0x1002a, 0x0)
        /usr/local/go/src/runtime/mheap.go:1047 +0x8a fp=0x7fff534ae7b0 sp=0x7fff534ae760 pc=0x42483a
runtime.(*mcentral).grow(0x2cffd80, 0x0)
        /usr/local/go/src/runtime/mcentral.go:256 +0x95 fp=0x7fff534ae7f8 sp=0x7fff534ae7b0 pc=0x417785
runtime.(*mcentral).cacheSpan(0x2cffd80, 0x7fd1a382d000)
        /usr/local/go/src/runtime/mcentral.go:106 +0x2ff fp=0x7fff534ae858 sp=0x7fff534ae7f8 pc=0x41728f
runtime.(*mcache).refill(0x7fd1a382d008, 0x2a)
        /usr/local/go/src/runtime/mcache.go:135 +0x86 fp=0x7fff534ae878 sp=0x7fff534ae858 pc=0x416d26
runtime.(*mcache).nextFree(0x7fd1a382d008, 0x2cf6e2a, 0x7fd1a382d008, 0x7fd1a382d000, 0x8)
        /usr/local/go/src/runtime/malloc.go:786 +0x88 fp=0x7fff534ae8b0 sp=0x7fff534ae878 pc=0x40b648
runtime.mallocgc(0x180, 0x190fb40, 0x1, 0x2d17920)
        /usr/local/go/src/runtime/malloc.go:939 +0x76e fp=0x7fff534ae950 sp=0x7fff534ae8b0 pc=0x40bf5e
runtime.newobject(0x190fb40, 0x4000)
        /usr/local/go/src/runtime/malloc.go:1068 +0x38 fp=0x7fff534ae980 sp=0x7fff534ae950 pc=0x40c368
runtime.malg(0x74ac0e00008000, 0x2d015f0)
        /usr/local/go/src/runtime/proc.go:3220 +0x31 fp=0x7fff534ae9c0 sp=0x7fff534ae980 pc=0x436871
runtime.mpreinit(...)
        /usr/local/go/src/runtime/os_linux.go:311
runtime.mcommoninit(0x2cf9240)
        /usr/local/go/src/runtime/proc.go:618 +0xc2 fp=0x7fff534ae9f8 sp=0x7fff534ae9c0 pc=0x430222
runtime.schedinit()
        /usr/local/go/src/runtime/proc.go:540 +0x74 fp=0x7fff534aea50 sp=0x7fff534ae9f8 pc=0x42feb4
runtime.rt0_go(0x7fff534aea88, 0x3, 0x7fff534aea88, 0x0, 0x0, 0x3, 0x7fff534aee07, 0x7fff534aee0f, 0x7fff534aee13, 0x0, ...)
        /usr/local/go/src/runtime/asm_amd64.s:195 +0x11a fp=0x7fff534aea58 sp=0x7fff534aea50 pc=0x458c4a
-- Promise Preston
bastion-host
kubectl
kubernetes

Similar Questions

Kibana with nginx ingress controller in Kubernetes
Readiness probe failed: Get http://10.32.1.71:80/setting s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Unable to add kubernetes-bionic main (Ubuntu 18.04) to apt repository
How do I connect Kubernetes Engine on GCP to an external Google Cloud Storage Bucket?
Kubernetes use External Services inside Pod

1 Answer

2/1/2022

I was able to fix the issue, it was due to insufficient memory (RAM) on the bastion host

Here's how I fixed it:

First I ran the command below to check the memory utilization:

free -m

which gave me the following output which confirmed that the insufficient memory (RAM):

              total        used        free      shared  buff/cache   available
Mem:           3.8G        3.7G         93M        988K         34M         27M
Swap:            0B          0B          0B

You can temporarily run the command below to free up little memory (RAM) space:

free && sync && echo 3 > /proc/sys/vm/drop_caches && free

What I did next was to upgrade the EC2 Instance type of the Bastion Host since it's an AWS EC2 Instance from t2.medium (4GB) to t2.large (8GB) using the steps stated here -Change the instance type. Basically, all you have to do is to:

  • choose Instance state > Stop instance
  • choose Actions > Instance settings > Change instance type
  • select a new instance type with higher capacity and choose Apply
  • choose Instance state > Start instance

Wait for a few minutes and everything should take effect. Please take note that your Public IP address for the Bastion Host might change after the upgrade.

Reference: Kubeflow deloyment error "fatal error: runtime: out of memory" #3803

That's all.

-- Promise Preston
Source: StackOverflow