So here is my case - working on a cluster with slurm and ubuntu, I have a .csh script that uses a for loop to spawn multiple jobs on the cluster (e.g. qsub job_1.csh, qsub job_2.csh and so on). each job_X.csh runs a single bash file sub_job.sh, with different inputs.
within each sub_job.sh, I try to source some file:
source full/path/to/file/env.csh
Out of a bunch of job_X.csh jobs that I spawn, some generate sub_job.sh run ok with no issues, but others fail on this sourcing line with:
job_X.csh: line 9: full/path/to/file/env.sh: No such file or directory
This seems to be related to the cluster/slurm. If I run this whole setting a few times, randomly some of the runs succeed and others fails, different every time. Any input would be great!
I tried to source all relevant env.sh files in the parent script of all job_X.csh files, but this did not propogate to the sub-job files.