I am running a cassandra pod on a Kubernetes worker node named kworker4 which has 16 GB RAM and no other pod is running in that worker node. I am using bitnami/cassandra.
The memory and CPU usage is like below in the Kubernetes cluster.
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%kmaster 225m 2% 1672Mi 10%kworker1 67m 0% 2499Mi 16%kworker2 28m 0% 1834Mi 11%kworker3 78m 0% 1487Mi 9%kworker4 359m 4% 1988Mi 12%kworker5 76m 0% 1244Mi 7%When I set maxHeapSize: "2048M" in values.yaml of bitnami/cassandra helm chart, the cassandra-0 pod is starting up.
But when I set maxHeapSize: "4096M" in values.yaml of bitnami/cassandra helm chart, the cassandra-0 pod is not starting up although the worker node kworker4 is running on a machine that has 16 GB RAM and from the top command it is found that memory utilization is very low without the cassandra-0 pod.
what is the issue?
I checked the syslog of Ubuntu and found the following:
Jul 10 13:44:47 kworker4 systemd[1]: Started Session 66 of User kadmin.Jul 10 13:49:37 kworker4 NetworkManager[938]: <info> [1720619377.9003] dhcp4 (wlp5s0): state changed new lease, address=10.227.25.233Jul 10 13:52:17 kworker4 kubelet[2861]: I0710 13:52:17.334274 2861 topology_manager.go:215] "Topology Admit Handler" podUID="45d3f529-6498-42a7-a26c-b255967b2533" podNamespace="default" podName="cassandra-0"Jul 10 13:52:17 kworker4 kubelet[2861]: E0710 13:52:17.334380 2861 cpu_manager.go:395] "RemoveStaleState: removing container" podUID="f2e22176-28e4-4a8f-87c1-2806a522832e" containerName="cassandra"Jul 10 13:52:17 kworker4 kubelet[2861]: I0710 13:52:17.334422 2861 memory_manager.go:354] "RemoveStaleState removing state" podUID="f2e22176-28e4-4a8f-87c1-2806a522832e" containerName="cassandra"Jul 10 13:52:17 kworker4 systemd[1]: Created slice libcontainer container kubepods-burstable-pod45d3f529_6498_42a7_a26c_b255967b2533.slice.Jul 10 13:52:17 kworker4 kubelet[2861]: I0710 13:52:17.519739 2861 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"empty-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d3f529-6498-42a7-a26c-b255967b2533-empty-dir\") pod \"cassandra-0\" (UID: \"45d3f529-6498-42a7-a26c-b255967b2533\") " pod="default/cassandra-0"Jul 10 13:52:17 kworker4 kubelet[2861]: I0710 13:52:17.519848 2861 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-cassandra-0\" (UniqueName: \"kubernetes.io/local-volume/data-cassandra-0\") pod \"cassandra-0\" (UID: \"45d3f529-6498-42a7-a26c-b255967b2533\") " pod="default/cassandra-0"Jul 10 13:52:17 kworker4 kubelet[2861]: I0710 13:52:17.621078 2861 operation_generator.go:664] "MountVolume.MountDevice succeeded for volume \"data-cassandra-0\" (UniqueName: \"kubernetes.io/local-volume/data-cassandra-0\") pod \"cassandra-0\" (UID: \"45d3f529-6498-42a7-a26c-b255967b2533\") device mount path \"/mnt/local-persistent-volumes/data-cassandra-0\"" pod="default/cassandra-0"Jul 10 13:52:17 kworker4 containerd[1054]: time="2024-07-10T13:52:17.946171772Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:cassandra-0,Uid:45d3f529-6498-42a7-a26c-b255967b2533,Namespace:default,Attempt:0,}"Jul 10 13:52:18 kworker4 networkd-dispatcher[957]: WARNING:Unknown index 39 seen, reloading interface listJul 10 13:52:18 kworker4 kernel: [111165.055341] weave: port 2(vethwepl9deef76) entered blocking stateJul 10 13:52:18 kworker4 kernel: [111165.055350] weave: port 2(vethwepl9deef76) entered disabled stateJul 10 13:52:18 kworker4 kernel: [111165.055607] device vethwepl9deef76 entered promiscuous modeJul 10 13:52:18 kworker4 NetworkManager[938]: <info> [1720619538.4870] manager: (vethwepg9deef76): new Veth device (/org/freedesktop/NetworkManager/Devices/37)Jul 10 13:52:18 kworker4 systemd-udevd[293361]: Using default interface naming scheme 'v249'.Jul 10 13:52:18 kworker4 NetworkManager[938]: <info> [1720619538.5216] manager: (vethwepl9deef76): new Veth device (/org/freedesktop/NetworkManager/Devices/38)Jul 10 13:52:18 kworker4 networkd-dispatcher[957]: ERROR:Unknown interface index 39 seen even after reloadJul 10 13:52:18 kworker4 networkd-dispatcher[957]: WARNING:Unknown index 39 seen, reloading interface listJul 10 13:52:18 kworker4 kernel: [111165.097012] eth0: renamed from vethwepg9deef76Jul 10 13:52:18 kworker4 systemd-networkd[921]: vethwepl9deef76: Link UPJul 10 13:52:18 kworker4 networkd-dispatcher[957]: ERROR:Unknown interface index 39 seen even after reloadJul 10 13:52:18 kworker4 systemd-networkd[921]: vethwepl9deef76: Gained carrierJul 10 13:52:18 kworker4 NetworkManager[938]: <info> [1720619538.5638] device (vethwepl9deef76): carrier: link connectedJul 10 13:52:18 kworker4 kernel: [111165.132876] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes readyJul 10 13:52:18 kworker4 kernel: [111165.132910] IPv6: ADDRCONF(NETDEV_CHANGE): vethwepl9deef76: link becomes readyJul 10 13:52:18 kworker4 kernel: [111165.132944] weave: port 2(vethwepl9deef76) entered blocking stateJul 10 13:52:18 kworker4 kernel: [111165.132947] weave: port 2(vethwepl9deef76) entered forwarding stateJul 10 13:52:18 kworker4 containerd[1054]: time="2024-07-10T19:22:18.687796519+05:30" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1Jul 10 13:52:18 kworker4 containerd[1054]: time="2024-07-10T19:22:18.687933851+05:30" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1Jul 10 13:52:18 kworker4 containerd[1054]: time="2024-07-10T19:22:18.687970510+05:30" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1Jul 10 13:52:18 kworker4 containerd[1054]: time="2024-07-10T19:22:18.688176283+05:30" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1Jul 10 13:52:18 kworker4 systemd[1]: Started libcontainer container 9deef764a0e07a31d7812c23efa211c36d9a74b470c5c07b9b6e0586dade5750.Jul 10 13:52:18 kworker4 containerd[1054]: time="2024-07-10T13:52:18.770045331Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:cassandra-0,Uid:45d3f529-6498-42a7-a26c-b255967b2533,Namespace:default,Attempt:0,} returns sandbox id \"9deef764a0e07a31d7812c23efa211c36d9a74b470c5c07b9b6e0586dade5750\""Jul 10 13:52:18 kworker4 containerd[1054]: time="2024-07-10T13:52:18.790190901Z" level=info msg="CreateContainer within sandbox \"9deef764a0e07a31d7812c23efa211c36d9a74b470c5c07b9b6e0586dade5750\" for container &ContainerMetadata{Name:cassandra,Attempt:0,}"Jul 10 13:52:19 kworker4 containerd[1054]: time="2024-07-10T13:52:19.007961508Z" level=info msg="CreateContainer within sandbox \"9deef764a0e07a31d7812c23efa211c36d9a74b470c5c07b9b6e0586dade5750\" for &ContainerMetadata{Name:cassandra,Attempt:0,} returns container id \"2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752\""Jul 10 13:52:19 kworker4 containerd[1054]: time="2024-07-10T13:52:19.008715302Z" level=info msg="StartContainer for \"2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752\""Jul 10 13:52:19 kworker4 systemd[1]: Started libcontainer container 2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752.Jul 10 13:52:19 kworker4 containerd[1054]: time="2024-07-10T13:52:19.141433055Z" level=info msg="StartContainer for \"2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752\" returns successfully"Jul 10 13:52:19 kworker4 avahi-daemon[935]: Joining mDNS multicast group on interface vethwepl9deef76.IPv6 with address fe80::5cae:7cff:fe8b:e880.Jul 10 13:52:19 kworker4 avahi-daemon[935]: New relevant interface vethwepl9deef76.IPv6 for mDNS.Jul 10 13:52:19 kworker4 systemd-networkd[921]: vethwepl9deef76: Gained IPv6LLJul 10 13:52:19 kworker4 avahi-daemon[935]: Registering new address record for fe80::5cae:7cff:fe8b:e880 on vethwepl9deef76.*.Jul 10 13:52:30 kworker4 systemd[1]: cri-containerd-2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752.scope: A process of this unit has been killed by the OOM killer.Jul 10 13:52:30 kworker4 kernel: [111176.800862] java invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=869Jul 10 13:52:30 kworker4 kernel: [111176.800866] CPU: 3 PID: 293719 Comm: java Not tainted 5.15.0-113-generic #123-UbuntuJul 10 13:52:30 kworker4 kernel: [111176.800868] Hardware name: LENOVO 20Q6S2WR01/20Q6S2WR01, BIOS R0ZET47W (1.25 ) 01/27/2021Jul 10 13:52:30 kworker4 kernel: [111176.800869] Call Trace:Jul 10 13:52:30 kworker4 kernel: [111176.800870] <TASK>Jul 10 13:52:30 kworker4 kernel: [111176.800872] show_stack+0x52/0x5cJul 10 13:52:30 kworker4 kernel: [111176.800875] dump_stack_lvl+0x4a/0x63Jul 10 13:52:30 kworker4 kernel: [111176.800878] dump_stack+0x10/0x16Jul 10 13:52:30 kworker4 kernel: [111176.800878] dump_header+0x53/0x228Jul 10 13:52:30 kworker4 kernel: [111176.800881] oom_kill_process.cold+0xb/0x10Jul 10 13:52:30 kworker4 kernel: [111176.800882] out_of_memory+0x106/0x2e0Jul 10 13:52:30 kworker4 kernel: [111176.800885] mem_cgroup_out_of_memory+0x13f/0x160Jul 10 13:52:30 kworker4 kernel: [111176.800888] try_charge_memcg+0x687/0x740Jul 10 13:52:30 kworker4 kernel: [111176.800889] charge_memcg+0x45/0xb0Jul 10 13:52:30 kworker4 kernel: [111176.800890] __mem_cgroup_charge+0x2d/0x90Jul 10 13:52:30 kworker4 kernel: [111176.800892] do_anonymous_page+0x114/0x3c0Jul 10 13:52:30 kworker4 kernel: [111176.800893] handle_pte_fault+0x20a/0x240Jul 10 13:52:30 kworker4 kernel: [111176.800895] __handle_mm_fault+0x405/0x6f0Jul 10 13:52:30 kworker4 kernel: [111176.800897] handle_mm_fault+0xd8/0x2c0Jul 10 13:52:30 kworker4 kernel: [111176.800898] do_user_addr_fault+0x1c9/0x670Jul 10 13:52:30 kworker4 kernel: [111176.800901] exc_page_fault+0x77/0x170Jul 10 13:52:30 kworker4 kernel: [111176.800902] asm_exc_page_fault+0x27/0x30Jul 10 13:52:30 kworker4 kernel: [111176.800905] RIP: 0033:0x7ff983e72220Jul 10 13:52:30 kworker4 kernel: [111176.800907] Code: 03 48 8d 05 96 57 8f 00 8b 13 89 10 e9 6e ff ff ff 90 0f 1f 84 00 00 00 00 00 55 48 39 f7 48 89 e5 73 12 0f 1f 80 00 00 00 00 <c6> 07 00 48 01 d7 48 39 fe 77 f5 5d c3 90 66 90 55 45 0f b6 c9 48Jul 10 13:52:30 kworker4 kernel: [111176.800908] RSP: 002b:00007ff9831fdc00 EFLAGS: 00010206Jul 10 13:52:30 kworker4 kernel: [111176.800909] RAX: 0000000000001000 RBX: 00007ff982c8f638 RCX: 00007ff9853208a3Jul 10 13:52:30 kworker4 kernel: [111176.800911] RDX: 0000000000001000 RSI: 0000000800000000 RDI: 00000007bee67000Jul 10 13:52:30 kworker4 kernel: [111176.800911] RBP: 00007ff9831fdc00 R08: 00000000ffffffff R09: 00000000f0000000Jul 10 13:52:30 kworker4 kernel: [111176.800912] R10: 0000000000000032 R11: 0000000000000246 R12: 0000000710000000Jul 10 13:52:30 kworker4 kernel: [111176.800913] R13: 00000000f0000000 R14: 0000000000000000 R15: 0000000800000000Jul 10 13:52:30 kworker4 kernel: [111176.800915] </TASK>Jul 10 13:52:30 kworker4 kernel: [111176.800916] memory: usage 3145728kB, limit 3145728kB, failcnt 56Jul 10 13:52:30 kworker4 kernel: [111176.800917] swap: usage 0kB, limit 9007199254740988kB, failcnt 0Jul 10 13:52:30 kworker4 kernel: [111176.800918] Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d3f529_6498_42a7_a26c_b255967b2533.slice:Jul 10 13:52:30 kworker4 kernel: [111176.800926] anon 3213541376Jul 10 13:52:30 kworker4 kernel: [111176.800926] file 90112Jul 10 13:52:30 kworker4 kernel: [111176.800926] kernel_stack 65536Jul 10 13:52:30 kworker4 kernel: [111176.800926] pagetables 6504448Jul 10 13:52:30 kworker4 kernel: [111176.800926] percpu 47296Jul 10 13:52:30 kworker4 kernel: [111176.800926] sock 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] shmem 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] file_mapped 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] file_dirty 90112Jul 10 13:52:30 kworker4 kernel: [111176.800926] file_writeback 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] swapcached 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] anon_thp 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] file_thp 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] shmem_thp 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] inactive_anon 3213463552Jul 10 13:52:30 kworker4 kernel: [111176.800926] active_anon 12288Jul 10 13:52:30 kworker4 kernel: [111176.800926] inactive_file 49152Jul 10 13:52:30 kworker4 kernel: [111176.800926] active_file 40960Jul 10 13:52:30 kworker4 kernel: [111176.800926] unevictable 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] slab_reclaimable 736952Jul 10 13:52:30 kworker4 kernel: [111176.800926] slab_unreclaimable 146752Jul 10 13:52:30 kworker4 kernel: [111176.800926] slab 883704Jul 10 13:52:30 kworker4 kernel: [111176.800926] workingset_refault_anon 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] workingset_refault_file 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] workingset_activate_anon 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] workingset_activate_file 0Jul 10 13:52:30 kworker4 kernel: [111176.800926] workingset_restore_anon 0Jul 10 13:52:30 kworker4 kernel: [111176.800927] Tasks state (memory values in pages):Jul 10 13:52:30 kworker4 kernel: [111176.800928] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj nameJul 10 13:52:30 kworker4 kernel: [111176.800928] [ 293427] 65535 293427 249 1 28672 0 -998 pauseJul 10 13:52:30 kworker4 kernel: [111176.800931] [ 293470] 1001 293470 1164985 787981 6496256 0 869 javaJul 10 13:52:30 kworker4 kernel: [111176.800933] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d3f529_6498_42a7_a26c_b255967b2533.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d3f529_6498_42a7_a26c_b255967b2533.slice/cri-containerd-2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752.scope,task=java,pid=293470,uid=1001Jul 10 13:52:30 kworker4 kernel: [111176.800941] Memory cgroup out of memory: Killed process 293470 (java) total-vm:4659940kB, anon-rss:3138004kB, file-rss:13920kB, shmem-rss:0kB, UID:1001 pgtables:6344kB oom_score_adj:869Jul 10 13:52:30 kworker4 kernel: [111176.800991] Tasks in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d3f529_6498_42a7_a26c_b255967b2533.slice/cri-containerd-2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752.scope are going to be killed due to memory.oom.group setJul 10 13:52:30 kworker4 kernel: [111176.800995] Memory cgroup out of memory: Killed process 293719 (java) total-vm:4659940kB, anon-rss:3138176kB, file-rss:14392kB, shmem-rss:0kB, UID:1001 pgtables:6344kB oom_score_adj:869Jul 10 13:52:30 kworker4 containerd[1054]: time="2024-07-10T13:52:30.229521112Z" level=info msg="TaskOOM event container_id:\"2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752\""Jul 10 13:52:30 kworker4 systemd[1]: cri-containerd-2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752.scope: Deactivated successfully.Jul 10 13:52:30 kworker4 systemd[1]: cri-containerd-2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752.scope: Consumed 1.333s CPU time.Jul 10 13:52:30 kworker4 systemd[1]: run-containerd-io.containerd.runtime.v2.task-k8s.io-2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752-rootfs.mount: Deactivated successfully.Jul 10 13:52:30 kworker4 containerd[1054]: time="2024-07-10T13:52:30.859741550Z" level=info msg="shim disconnected" id=2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752 namespace=k8s.ioJul 10 13:52:30 kworker4 containerd[1054]: time="2024-07-10T13:52:30.859867500Z" level=warning msg="cleaning up after shim disconnected" id=2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752 namespace=k8s.ioJul 10 13:52:30 kworker4 containerd[1054]: time="2024-07-10T13:52:30.859893980Z" level=info msg="cleaning up dead shim" namespace=k8s.ioJul 10 13:52:31 kworker4 kubelet[2861]: I0710 13:52:31.580868 2861 scope.go:117] "RemoveContainer" containerID="2cb8ea27c355aae5a717ce8c7ffab53f627ac0ce73f7349a0a0c009c6bb80752"