Bug 812617 - stop service times out permenantly
stop service times out permenantly
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: rsyslog (Show other bugs)
16
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Tomas Heinrich
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-04-15 09:09 EDT by Harald Reindl
Modified: 2016-09-20 00:50 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-13 21:27:01 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
rsyslog.conf with obfuscated mysql password (9.51 KB, text/plain)
2012-04-28 10:39 EDT, Harald Reindl
no flags Details

  None (edit)
Description Harald Reindl 2012-04-15 09:09:03 EDT
am i the only one seeing rsyslog on many machines hangig until timeout
if the service is stopped - including reboot/shutdown)

yet updated to rsyslog-5.8.10-1.fc16.x86_64, the same

systemd[1]: rsyslog.service stopping timed out. Killing.
systemd[1]: rsyslog.service: main process exited, code=killed, status=9
systemd[1]: Unit rsyslog.service entered failed state.

i had to clone the systemd-unit because i can not wait
5 minutes for rsyslog on each reboot / shutdown especially
on production servers

additionally i restored the missing Reload-Action
________________________

[root@srv-rhsoft:~]$ cat /etc/systemd/system/rsyslog.service
[Unit]
Description=System Logging Service

[Service]
EnvironmentFile=-/etc/sysconfig/rsyslog
ExecStartPre=/bin/systemctl stop systemd-kmsg-syslogd.service
ExecStart=/sbin/rsyslogd -n $SYSLOGD_OPTIONS
ExecReload=/usr/bin/killall -s SIGHUP rsyslogd
TimeoutSec=2
Sockets=syslog.socket
StandardOutput=null
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target
Alias=syslog.service
Comment 1 Harald Reindl 2012-04-16 04:59:06 EDT
i randomly found out today that the only machine out of 25 which does not use rsyslog-mysql is not involved - so i guess this is why nobody noticed this until now

$ModLoad ommysql
$WorkDirectory /var/spool/rsyslog                   # default location for work (spool) files
$ActionQueueType LinkedList                         # use asynchronous processing
$ActionQueueFileName dbq                            # set file name, also enables disk mode
$ActionResumeRetryCount -1                          # infinite retries on insert failure
*.info;mail.none;authpriv.none;cron.none            :ommysql:localhost,dbmail,vm_syslog,*****
mail.*                                              :ommysql:localhost,dbmail,vm_syslog,*****
Comment 2 Tomas Heinrich 2012-04-16 08:47:54 EDT
(In reply to comment #1)
> i randomly found out today that the only machine out of 25 which does not use
> rsyslog-mysql is not involved - so i guess this is why nobody noticed this
> until now

> $ActionResumeRetryCount -1                          # infinite retries on

I guess this is the problem. It basically means infinite retries (until the daemon is killed).

> additionally i restored the missing Reload-Action
> ExecReload=/usr/bin/killall -s SIGHUP rsyslogd

This won't work. To reload configuration, the daemon needs to be restarted. This merely reopens log files (used for log rotation).
http://blog.gerhards.net/2008/10/new-rsyslog-hup-processing.html
http://www.rsyslog.com/doc/v5compatibility.html
Comment 3 Harald Reindl 2012-04-16 09:07:10 EDT
>> $ActionResumeRetryCount -1                          # infinite retries on
>
> I guess this is the problem. It basically means infinite retries (until the
> daemon is killed).

simply no

after uncomment the line it fails also
systemd[1]: rsyslog.service stopping timed out. Killing

even if, this line is needed because the full comment says "# infinite retries on insert failure" and without you will lose messages while restarting mysqld or reboot the machine while mysqld starts later as syslogd

please try it out - "rsyslog-mysql" does introduce this reboot/restart-problem with large timeouts since the first day of Fedora 15 while we are using this since years from F9-F14
Comment 4 Tomas Heinrich 2012-04-20 08:43:49 EDT
(In reply to comment #3)
> please try it out - "rsyslog-mysql" does introduce this reboot/restart-problem
> with large timeouts since the first day of Fedora 15 while we are using this
> since years from F9-F14

I've been able to reproduce the hang with ommysql, even segfaults in some cases.
This seems to happen when using the same ommysql action multiple times.
I'll keep you posted.
Comment 5 Harald Reindl 2012-04-20 08:55:38 EDT
thank you for your feedback!

i need the same action twice because the database for all mail-logs is on another server as all other syslog (the example above is simplified)

finally we are using in our self-developed admin-panel this tables for filtering and this is used currently for 25 machines with a insert-only user and so syslog can keep closed in the whole network

wondering that we are the only one doing so :-)
Comment 6 Tomas Heinrich 2012-04-24 11:35:24 EDT
(In reply to comment #5)
> wondering that we are the only one doing so :-)

I agree that it's weird this didn't show up earlier.
Here are new packages to try out. They contain a simple patch that may fix this issue.
http://theinric.fedorapeople.org/rsyslog-5.8.10-2.fc16.x86_64/
Comment 7 Harald Reindl 2012-04-24 11:41:07 EDT
sorry, but no success

[root@rh:~]$ rpm -qa | grep rsyslog
rsyslog-5.8.10-2.fc16.x86_64
rsyslog-mysql-5.8.10-2.fc16.x86_64

[root@rh:~]$ dmesg -c
systemd[1]: rsyslog.service stopping timed out. Killing.
systemd[1]: rsyslog.service: main process exited, code=killed, status=9
systemd[1]: Unit rsyslog.service entered failed state.
[root@rh:~]$ dmesg -c
[root@rh:~]$ systemctl restart rsyslog.service
[root@rh:~]$ dmesg -c
systemd[1]: rsyslog.service stopping timed out. Killing.
systemd[1]: rsyslog.service: main process exited, code=killed, status=9
systemd[1]: Unit rsyslog.service entered failed state.
[root@rh:~]$ systemctl restart rsyslog.service
[root@rh:~]$ dmesg -c
systemd[1]: rsyslog.service stopping timed out. Killing.
systemd[1]: rsyslog.service: main process exited, code=killed, status=9
systemd[1]: Unit rsyslog.service entered failed state.
Comment 8 Tomas Heinrich 2012-04-24 12:09:43 EDT
With the new packages, I don't experience any segfaults, but the shutdown still takes some time (~10 sec). Could you please try to increase TimeoutSec to something like 30 sec?

If this doesn't help, try running the daemon in the debug mode from the shell:
rsyslogd -c 5 -dnf /etc/rsyslog.conf
Send a SIGTERM to it or press ctrl-c and see what happens.
If nothing interesting comes out of it, please provide some more specific configuration with which you experience problems, because I can't reproduce any issues right now.
Comment 9 Harald Reindl 2012-04-28 10:35:46 EDT
timeout of 30 seconds is way too much

keep in mind that most servic es are still stopped on reboot/shutdown while rsyslog is stopped and so it would re-introduce the problem i solve with the timeout of 2 seconds
_______________________________________________

one interesting output on debug:

3333.267824954:7fd4d4a31700: error 14 unlinking '(null)' - ignored: Bad address
Error in my_thread_global_end(): 1 threads didn't exit
Comment 10 Harald Reindl 2012-04-28 10:39:23 EDT
Created attachment 580977 [details]
rsyslog.conf with obfuscated mysql password

attached the complete "rsyslog.conf" from my development-machine
below my complete systemd-unit


cat /etc/systemd/system/rsyslog.service
[Unit]
Description=System Logging Service

[Service]
EnvironmentFile=-/etc/sysconfig/rsyslog
ExecStartPre=/bin/systemctl stop systemd-kmsg-syslogd.service
ExecStart=/sbin/rsyslogd -n $SYSLOGD_OPTIONS
ExecReload=/usr/bin/killall -s SIGHUP rsyslogd
TimeoutSec=2
Sockets=syslog.socket
StandardOutput=null
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target
Alias=syslog.service
Comment 11 Tomas Heinrich 2012-05-03 07:31:15 EDT
(In reply to comment #10)
> attached the complete "rsyslog.conf" from my development-machine
> below my complete systemd-unit

Sorry, I'm currently out of time, will try to reproduce it later.

The 30 sec timeout was only meant for testing to see if the symptoms still appear.

If you want, you can try the sources from the upstream repository - it has some additional bugfixes, which possibly can help:
http://git.adiscon.com/?p=rsyslog.git;a=shortlog;h=refs/heads/v5-stable
Comment 12 Fedora End Of Life 2013-01-16 19:34:15 EST
This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 13 Fedora End Of Life 2013-02-13 21:27:04 EST
Fedora 16 changed to end-of-life (EOL) status on 2013-02-12. Fedora 16 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.