/var/lib/docker/overlay2/*/merged take too much space than it should be

problem descriptions:

In my system, /var/lib/docker/overlay2/*/merged take too much space than it should be and the detail looks like below.

root@sriov-TaiShan-200:~# df -h --totalFilesystem          Size  Used Avail Use% Mounted onudev                 63G     0   63G   0% /devtmpfs                13G  6.2M   13G   1% /run/dev/nvme0n1p2      2.9T  2.4T  329G  89% /tmpfs                63G     0   63G   0% /dev/shmtmpfs               5.0M     0  5.0M   0% /run/locktmpfs                63G     0   63G   0% /sys/fs/cgroup/dev/loop0          128K  128K     0 100% /snap/bare/5/dev/loop1           49M   49M     0 100% /snap/core18/2406/dev/loop2           49M   49M     0 100% /snap/core18/2349/dev/loop3           58M   58M     0 100% /snap/core20/1437/dev/loop5          229M  229M     0 100% /snap/gnome-3-38-2004/100/dev/loop6           73M   73M     0 100% /snap/lxd/23043/dev/loop7          198M  198M     0 100% /snap/gnome-3-34-1804/73/dev/nvme0n1p1      512M  5.2M  507M   1% /boot/efi/dev/loop8           66M   66M     0 100% /snap/gtk-common-themes/1519/dev/loop9           49M   49M     0 100% /snap/snap-store/545/dev/loop10         229M  229M     0 100% /snap/gnome-3-38-2004/88/dev/loop11          39M   39M     0 100% /snap/snapd/15541/dev/loop14         198M  198M     0 100% /snap/gnome-3-34-1804/75/dev/loop15          73M   73M     0 100% /snap/lxd/23003/dev/loop16          53M   53M     0 100% /snap/snap-store/559//10.67.93.97/work  916G  816G  100G  90% /mnt/dev/loop17          82M   82M     0 100% /snap/gtk-common-themes/1534/dev/loop13          39M   39M     0 100% /snap/snapd/15909/dev/loop12          58M   58M     0 100% /snap/core20/1498tmpfs                13G   16K   13G   1% /run/user/0tmpfs               126G   12K  126G   1% /var/lib/kubelet/pods/12baa83c-6696-43f3-b060-85ba620867aa/volumes/kubernetes.io~projected/kube-api-access-j26q8overlay             2.9T  2.4T  329G  89% /var/lib/docker/overlay2/150f4adf713b9882cd39e5580204040f20dcebf20205c9c62b55a85ac416da89/mergedshm                  64M     0   64M   0% /var/lib/docker/containers/9b02ed0d68aa3c24531b1e2852861a202b36b9d7d08645ee170035a3eab48747/mounts/shmoverlay             2.9T  2.4T  329G  89% /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d/mergedtotal               9.9T  8.0T  1.4T  86% -

root@sriov-TaiShan-200:~# docker system dfTYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLEImages          2         2         109.7MB   0B (0%)Containers      2         2         2.294kB   0B (0%)Local Volumes   0         0         0B        0BBuild Cache     0         0         0B        0B

root@sriov-TaiShan-200:~# du -h --max-depth=1 /var/lib/docker/overlay2 | sort -rh | head -25223M    /var/lib/docker/overlay2111M    /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d68M /var/lib/docker/overlay2/6b7c00ae09c4df6f35e65114513bd6216c2436e62ae26c3b91b4c2ab6073466242M /var/lib/docker/overlay2/94b81b09cc4a7ca3ead72888264e3cff25af77d2fc00334847aa2c3bde225e8a1.6M    /var/lib/docker/overlay2/65ca885064b7df65728ea2864d3f633f035d4b9f3348ae4bcdc8c926157b9552528K    /var/lib/docker/overlay2/150f4adf713b9882cd39e5580204040f20dcebf20205c9c62b55a85ac416da89488K    /var/lib/docker/overlay2/5e344fbdc03c21013a38b1a82987d7e9094b5d13fe96d0f89f6e31a096f51c9c48K /var/lib/docker/overlay2/150f4adf713b9882cd39e5580204040f20dcebf20205c9c62b55a85ac416da89-init36K /var/lib/docker/overlay2/l32K /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d-init

Things i tried:

1. extend lv /dev/ubuntu-vg/ubuntu-lv size but no lucky
2. reinstall the docker with below steps but no lucky

apt-get purge docker-cerm -rf /var/lib/dockerrm -rf /etc/dockerapt-get remove docker docker-engine docker.io containerd runc/etc/apt/sources.list.d/docker.list > /dev/nullapt updatemkdir -p /etc/apt/keyringscurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpgapt updateapt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

3. docker system prune -a but no luck

root@sriov-TaiShan-200:~# docker system prune -aWARNING! This will remove:  - all stopped containers  - all networks not used by at least one container  - all images without at least one container associated to them  - all build cacheAre you sure you want to continue? [y/N] yTotal reclaimed space: 0B

updates

It seems that docker wasn't using that much disk space. Through the du result, it is other things that take the space. However the overlay 2.9T 2.4T 329G 89% /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d/merged is quite confusing.

root@sriov-TaiShan-200:# du -h --max-depth=1 / --exclude=mnt --exclude=proc --exclude=home | sort -rh2.3T    /1.6T    /var686G    /root15G /usr6.1G    /snap3.6G    /x86_64-linux-gnu1.2G    /boot88M /opt16M /etc6.2M    /run92K /tmp16K /lost+found12K /media4.0K    /srv4.0K    /cdrom0   /sys0   /dev