How to correct measure memory usage of multi-threading in R language by shell?

I have a shell script to monitor system resource usage and send alerts. Basically, it relies on the following commands:

ps -u "$user" --no-headers -o pmem | awk '{sum += $1} END {print sum}'

In most cases, it works well, but recently I noticed some issues.

For example, run with a clean rebooted system to minimize the effect of other processes:

# Rlibrary(data.table)library(parallel)set.seed(123)n_rows <- 1e7  n_cols <- 1000## memory black hole!!! change it if necessary, however, you need to require a high level of memory usage (about 1/30 to 1/10 of your usable memory) to achieve significant results in the subsequent testsdt <- data.table(matrix(runif(n_rows * n_cols), nrow = n_rows, ncol = n_cols))memory_usage <- object.size(dt)format(memory_usage, units = "auto")worker_function <- function(data) {  row_sums <- rowSums(data)  ## wait about 30s and run shell command to test when process sleep  Sys.sleep(1000)  return(row_sums)}n_threads <- 10results <- mclapply(1:n_threads, function(x) worker_function(dt), mc.cores = n_threads)

check memory status by top

top

we can get memory usage precent by sum of the MEM% col, or by following command:

ps -u "$user" --no-headers -o pmem | awk '{sum += $1} END {print sum}'# 113.9

In my system (1.5 TiB memory and 2.0 GiB swap) you will find this value larger than 100, but if you check memory by free -h, you will get:

only 645/1500 Gib be used, it less than 50%