currently we use rsyslog to collect webservers logs in one VM.Log sync always hangs indefinitely after log rotation script execution on Ubuntu22.
we have this rsyslog version and config file:
rsyslogd: version 8.32.0
$ModLoad imfile$InputFilePollInterval 10$PrivDropToGroup adm$InputFileName /app/appdirectory/logs/ourlogs.out$InputFileTag ourlogs$InputFileStateFile /var/spool/ourlogs$InputFileSeverity info$InputFileFacility local1$InputRunFileMonitor$InputFilePersistStateInterval 1000*.* @@our.monitoringserver.com:x
Our rotation script was set to run daily and on web server restart to check if the logsare > 100 Mo if so the following shell script is executed:
if [ -f ourlogs.out ]; then cp -f ourlogs.out ourlogs.out.$DATE && cat /dev/null > ourlogs.out gzip ourlogs.out.$DATE mv ourlogs.out.$DATE.gz $backupDir/fi
Log sync always hangs indefinitely after log rotation script execution untill we restart rsyslog.
After restarting rsyslog everything works as expected.
The problem arises as we are moving from Ubuntu20 to Ubuntu22, restarting rsyslog doesn't solve the issue anymore on Ubuntu22.
Our investigation led to: rsyslog waits till the new log is larger than the old log file to sync again, so it's a problem of how rsyslog keeps track of the new logs.
to solve this we have moved to the following log rotation script:
if [ -f ourlogs.out ]; then **mv -f ourlogs.out ourlogs.out.$DATE && touch ourlogs.out** gzip ourlogs.out.$DATE mv ourlogs.out.$DATE.gz $backupDir/fi
My question is what are the risks come with this solution, could we lose logs between the mv command and the touch command?
Lastly we have chosen this method rather than logrotate / output channels because it's easier to implement, but if there are risks with the proposed solution we will definitely migrate to what's recommended.
Any recommendation are welcome.