Description of problem:

When forwarding logs to a remote server and that server is not reachable for some time, logs are queued to disk.
Upon remote server being reachable again, the logs are transmitted.

However, if rsyslog crashes for some reason or system is abruptly restarted (i.e. without rsyslog shutting down gracefully), upon restart of rsyslog, the disk queue files are not removed, because there is no associated ".qi" file.


Version-Release number of selected component (if applicable):

rsyslog-5.8.10-10.el6_6.x86_64 but affects also other newer releases (7.x, 8.x)


How reproducible:

Always


Steps to Reproduce:
1. Set up forwarding on vm-rsyslog69-client to vm-rsyslog74-server

(vm-rsyslog69-client) Excerpt of /etc/rsyslog.conf

# Disk Assisted Config
$ActionQueueType LinkedList
$WorkDirectory /var/lib/rsyslog
$ActionQueueSize 100000
$ActionQueueFileName diskqueue
$ActionQueueMaxDiskSpace 1G
$ActionQueueSaveOnShutdown on
$ActionResumeRetryCount -1
$ActionResumeInterval 10
$ActionQueueLowWatermark 200
$ActionQueueHighWatermark 600
*.info @@vm-rsyslog74-server:514

(vm-rsyslog74-server) Excerpt of /etc/rsyslog.conf

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514


2. Generate logs from vm-rsyslog69-client

(vm-rsyslog69-client) # i=1; while :; do logger "HEY $(printf "%05d" $i) xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; sleep 0.1; i=$(($i+1)); done


3. After some time, stop rsyslog on remote server

(vm-rsyslog74-server) # systemctl stop rsyslog


4. Wait for vm-rsyslog69-client to queue logs on disk

(vm-rsyslog69-client) # watch ls -l /var/lib/rsyslog
total 216
-rw-------. 1 root root 217799 Oct 20 12:06 diskqueue.00000001


5. Start rsyslog on remote server

(vm-rsyslog74-server) # systemctl start rsyslog


6. Stop generating logs from vm-rsyslog69-client 

(vm-rsyslog69-client) ^C


7. Kill rsyslog or reboot system abruptly


Additional info:

Since there is no ".qi" file, rsyslog will not remove the unusable disk queue files, consuming disk space.
Because the ".qi" file is only created when rsyslog gracefully shut downs, logs will not be sent to remote server (in case remote server was unreachable while the client system rebooted abruptly).

As an improvment, the ".qi" file may be created as soon as disk queueing happens, and removed once disk queue gets empty.
By doing so, upon start of rsyslog, rsyslog may safely delete queue files which do not have a corresponding ".qi" file.