Rotating log files which are being held by an open file descriptor

Some processes keep a file descriptor on their log files open permanently. This can lead to strange results when the logs are being rotated.

Normally log rotation programs move the old log file, compress it and create a new one. But the process still wants to write to the old log file. What happens is that the new log file remains empty. The process can still write to the old file descriptor but this data cannot be written to the old log file because it is compressed. The data is being written somewhere onto the disk and the free disk space decreases. You have no access to this data any more. No object in the file system represents this data. So disk space is lost and the logging data is lost as well.

Many daemons accept a SIGHUP signal. It can be used to trigger a reconfiguration of the daemon and a reopening of the file descriptors.

kill -HUP PID

The log rotation program can send this signal and rotate the log file without problems. But other daemons just ignore this signal and they cannot be triggered to reopen their file descriptors.
One solution for this is to copy the log file (instead of moving it) so that the file descriptor still points to the current log file and write /dev/null into the current log file. However there is a small period between the copying and the writing of /dev/null which may cause some logging data to get lost.

Here's a little script that helps to automate things. The file is being rotated and compressed if it has exceeded the maximum size.

#!/usr/local/bin/bash

# needs one argument which is maxsize in MB

(( maxsize= $1 * 1024 * 1024 ))
dir=/var/log
file=test.log
size=`ls -la $dir/$file  | awk '{print $5}'`
if [ $size -gt $maxsize ]
then
date=`date +%y%m%d-%H%M%S`
tar cfvzp /var/log/"$file.$date".tgz "$dir/$file"
cat /dev/null > "$dir/$file"
fi

Much easier is the usage of the RPM package 'logrotate' which is able to copy and truncate old log files instead of moving them. On Gentoo just do 'emerge logrotate' and put the 'copytruncate' directive in '/etc/logrotate.conf'.