Bug 699198 - rsyslogd not enabled after upgrade -- no logging in /var/log/messages
rsyslogd not enabled after upgrade -- no logging in /var/log/messages
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: rsyslog (Show other bugs)
15
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Tomas Heinrich
Fedora Extras Quality Assurance
RejectedBlocker, AcceptedNTH
:
: 702425 769918 (view as bug list)
Depends On: 702378
Blocks: F15-accepted/F15FinalFreezeExcept
  Show dependency treegraph
 
Reported: 2011-04-24 03:08 EDT by cam
Modified: 2015-12-02 05:54 EST (History)
19 users (show)

See Also:
Fixed In Version: rsyslog-5.7.9-3.fc15
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-05-19 01:09:09 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
debug output mentioned in wiki (431.03 KB, text/plain)
2011-04-27 03:09 EDT, cam
no flags Details

  None (edit)
Description cam 2011-04-24 03:08:29 EDT
Description of problem:
Since 10 days ago I have seen no log messages accrue in /var/log/messages. Other log files are updated (eg. /var/log/gdm)

Version-Release number of selected component (if applicable):
rsyslog-5.7.9-2.fc15.i686


How reproducible:
Not sure what happened but my machine is in this state

Steps to Reproduce:
1.unsure

Actual results:
logging stops

Expected results:
logging continues

Additional info:

[root@newt ~]# ls -ltr /var/log/messages*
-rw-------. 1 root root  891564 Mar 27 12:16 /var/log/messages-20110327
-rw-------. 1 root root  891295 Apr  3 08:16 /var/log/messages-20110403
-rw-------. 1 root root 1410961 Apr 11 08:02 /var/log/messages-20110411
-rw-------. 1 root root  329207 Apr 14 08:20 /var/log/messages-20110423
-rw-------. 1 root root       0 Apr 23 09:20 /var/log/messages


Last entries are:


Apr 14 08:20:19 newt rtkit-daemon[1656]: Demoting known real-time threads.
Apr 14 08:20:19 newt rtkit-daemon[1656]: Successfully demoted thread 1892 of process 1887 (/usr/bin/pulseaudio).
Apr 14 08:20:19 newt init: Disconnected from system bus
Apr 14 08:20:19 newt rtkit-daemon[1656]: Successfully demoted thread 1891 of process 1887 (/usr/bin/pulseaudio).
Apr 14 08:20:19 newt rtkit-daemon[1656]: Successfully demoted thread 1887 of process 1887 (/usr/bin/pulseaudio).
Apr 14 08:20:19 newt rtkit-daemon[1656]: Demoted 3 threads.
Apr 14 08:20:19 newt nm-dispatcher.action: Disconnected from the system bus, exiting.
Apr 14 08:20:19 newt rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Apr 14 08:20:19 newt kernel: [ 1579.968633] type=1305 audit(1302765619.679:28838): audit_pid=0 old=1736 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
Apr 14 08:20:19 newt auditd[1736]: The audit daemon is exiting.
Apr 14 08:20:20 newt cpuspeed: Disabling ondemand cpu frequency scaling governor
Apr 14 08:20:20 newt kernel: [ 1581.152818] type=1701 audit(1302765620.863:28839): auid=500 uid=500 gid=500 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=2895 comm="gvfs-gdu-volume" sig=6
Apr 14 08:20:21 newt kernel: Kernel logging (proc) stopped.
Apr 14 08:20:21 newt rsyslogd: [origin software="rsyslogd" swVersion="4.6.3" x-pid="1093" x-info="http://www.rsyslog.com"] exiting on signal 15.
Comment 1 Stylianos Modes 2011-04-25 20:48:30 EDT
I see this as well, in fact, rsyslog has never worked for me since I upgraded to F15. Last messages are the normal shutdown on F14 while running the preupgrade and then nothing.

Version:
rsyslog.x86_64      5.7.9-2.fc15

After boot, I get:

# service rsyslog status
rsyslog.service - System Logging Service
          Loaded: loaded (/lib/systemd/system/rsyslog.service)
          Active: inactive (dead)
          CGroup: name=systemd:/system/rsyslog.service

If I start the service manually it then works just fine (and includes all messages since last boot).

# chkconfig --list | grep syslog

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overriden by native
      systemd configuration.

rsyslog         0:off   1:off   2:on    3:on    4:on    5:on    6:off

I'm not familiar with systemd to understand if rsyslog is even trying to start at boot.
Comment 2 Tomas Heinrich 2011-04-26 10:01:46 EDT
(In reply to comment #0)
Is the rsyslogd process still running?
Do you have selinux in enforcing mode?
Do you have selinux-policy updated to the latest version?
Are there any relevant AVCs in /var/log/audit/audit.log?
  $ ausearch -m avc -c rsyslog
What's the output of
  $ systemctl status rsyslog.service
Does restarting the service help?
Is rsyslog configured to start with systemd?
  $ ls /etc/systemd/system/*.wants/rsyslog.service

(In reply to comment #1)
The same questions here.
Here's some info re systemd troubleshooting:
http://fedoraproject.org/wiki/How_to_debug_Systemd_problems
Comment 3 cam 2011-04-27 03:05:51 EDT
(In reply to comment #2)
> (In reply to comment #0)
> Is the rsyslogd process still running?
No, even after reboots

> Do you have selinux in enforcing mode?
SELINUX=enforcing

> Do you have selinux-policy updated to the latest version?
yes, regular yum updates and the first thing I did on seeing this problem was to relabel everything, just in case.

> Are there any relevant AVCs in /var/log/audit/audit.log?
>   $ ausearch -m avc -c rsyslog
<no matches>
(interesting command, I'll remember it)

> What's the output of
>   $ systemctl status rsyslog.service
rsyslog.service - System Logging Service
	  Loaded: loaded (/lib/systemd/system/rsyslog.service)
	  Active: inactive (dead)
	  CGroup: name=systemd:/system/rsyslog.service


> Does restarting the service help?
Only until the next boot, it seems

> Is rsyslog configured to start with systemd?
>   $ ls /etc/systemd/system/*.wants/rsyslog.service
no such file, so I suppose no

> Here's some info re systemd troubleshooting:
> http://fedoraproject.org/wiki/How_to_debug_Systemd_problems

It looks like it's failing to start somehow at boot time, I'm not sure where to go from here.
Comment 4 cam 2011-04-27 03:09:45 EDT
Created attachment 495118 [details]
debug output mentioned in wiki
Comment 5 Tomas Heinrich 2011-04-27 08:24:42 EDT
The symptoms of this bug can probably be easily remedied by running
  # systemctl enable rsyslog.service

The cause seems to be that the rsyslog service is not enabled when upgrading from a version that uses a SysV script (/etc/rc.d/init.d/rsyslog)
to a version that uses a systemd unit file (/lib/systemd/system/rsyslog.service),
even if the service was enabled in chkconfig.

Not sure what's the right way to fix this. Lennart?
Comment 6 Stylianos Modes 2011-04-27 14:08:58 EDT
(In reply to comment #2)
> (In reply to comment #1)
> The same questions here.
> Here's some info re systemd troubleshooting:
> http://fedoraproject.org/wiki/How_to_debug_Systemd_problems
My results are exactly the same as those in comment #3.

(In reply to comment #5)
> The symptoms of this bug can probably be easily remedied by running
>   # systemctl enable rsyslog.service
Thank you for the workaround, service starts on boot now and the setting persists across reboot cycles.
Comment 7 cam 2011-04-27 17:35:42 EDT
(In reply to comment #5)
> The symptoms of this bug can probably be easily remedied by running
>   # systemctl enable rsyslog.service
> 
> The cause seems to be that the rsyslog service is not enabled when upgrading
> from a version that uses a SysV script (/etc/rc.d/init.d/rsyslog)
> to a version that uses a systemd unit file
> (/lib/systemd/system/rsyslog.service),
> even if the service was enabled in chkconfig.
> 
> Not sure what's the right way to fix this. Lennart?

Fascinating... ideally we would have some way to detect that something is missing before we come to use it... I'll bet many system services might be missing like this after upgrade. No wonder it boots faster ;-) Is there a mechanical way to audit it?
Comment 8 Lennart Poettering 2011-04-27 18:43:09 EDT
The code to handle the upgrade is actually existant in the spec file.

http://pkgs.fedoraproject.org/gitweb/?p=rsyslog.git;a=blob;f=rsyslog.spec;h=b5102d3eccf4d942f1c8fc3e8bb433545f2b4dc4;hb=refs/heads/f15/master#l219

If you run that excerpt manually, does it work then for you?
Comment 9 cam 2011-04-28 02:01:33 EDT
(In reply to comment #8)
> The code to handle the upgrade is actually existant in the spec file.
> 
> http://pkgs.fedoraproject.org/gitweb/?p=rsyslog.git;a=blob;f=rsyslog.spec;h=b5102d3eccf4d942f1c8fc3e8bb433545f2b4dc4;hb=refs/heads/f15/master#l219
> 
> If you run that excerpt manually, does it work then for you?

Sorry, I'm not sure exactly what code to run or if this is now a valid context to run it in - having manually applied the remedy in comment #5. I take it you mean the stuff in the %post, which just seems to do 

 chkconfig --add rsyslog

there is also some conditional stuff in there that says it's for install, not upgrade - I take it in my case it would have been an upgrade. I still have the upgrade.log and upgrade.log.syslog files if that would help... The upgrade log mentions chkconfig and rsyslog in the correct order:

 Upgrading chkconfig-1.3.51-1.fc15.i686
 [...]
 Upgrading rsyslog-5.7.9-2.fc15.i686
Comment 10 Tomas Heinrich 2011-04-28 08:22:09 EDT
(In reply to comment #9)
I think the code in question is this:
 219 %triggerun -- rsyslog < 5.7.8-1
 220 if /sbin/chkconfig --level 3 rsyslog ; then
 221         /bin/systemctl --no-reload enable rsyslog.service >/dev/null 2>&1 || :
 222 fi

For me, running it manually works.

With rpm, I get this:

# systemctl disable rsyslog.service
rm '/etc/systemd/system/multi-user.target.wants/rsyslog.service'
# rpm --force -U rsyslog-4.6.3-2.fc15.x86_64.rpm
# chkconfig --level 2345 rsyslog on
# rpm -vv -U rsyslog-5.7.9-2.fc15.x86_64.rpm
<...>
D:     0    0    0    1   +rsyslog-5.7.9-2.fc15.x86_64
D:     1    0    0    1   -rsyslog-4.6.3-2.fc15.x86_64
<...>
D: %post(rsyslog-5.7.9-2.fc15.x86_64): waitpid(5338) rc 5338 status 0
D:  read h#     317 Header SHA1 digest: OK (bd3fe00e9e67d3d431adeb8882827344b800840a)
D: ========== +++ rsyslog-4.6.3-2.fc15 x86_64-linux 0x2
D:     erase: rsyslog-4.6.3-2.fc15 has 135 files
D: (null): scriptlet start
D: (null): execv(/bin/sh) pid 5343
+ '[' -f /var/lock/subsys/rsyslogd ']'
+ mv /var/lock/subsys/rsyslogd /var/lock/subsys/rsyslog
D: (null): waitpid(5343) rc 5343 status 0
D: %preun(rsyslog-4.6.3-2.fc15.x86_64): scriptlet start
<...>

Looks like only the second 'triggerun' gets run and the third one doesn't.
Comment 11 TR Bentley 2011-05-02 06:40:06 EDT
Having the same issues with no logging but with a twist:
systemctl status rsyslog.service fails with a 
Failed to get D-Bus connection: Failed to connect to socket
/org/freedesktop/systemd1/private: Connection refused

* Systemadm also fails with this error.

I have no dmesg or messages logged.

Chkconfig --add rsyslog.service did not error.

I have not lines in my grub.conf to tell systemd to start and I thought there was meant to be a line!

I have an F14 virtual machine upgraded to F15 Beta1 and then patched upto date using package manager.
Comment 12 Gene Snider 2011-05-02 12:27:04 EDT
I just ran into this bug after updating from F14 to F15.  Is there some way to check for other services that got lost during the update?

Thanks,
Gene
Comment 13 cam 2011-05-04 17:29:33 EDT
(In reply to comment #12)
> I just ran into this bug after updating from F14 to F15.  Is there some way to
> check for other services that got lost during the update?
> 
> Thanks,
> Gene

I completely agree, an automated method that will root out all of these problems is desperately needed. It seems there are several cases of different services not enabled after upgrade (when they should have been). Some cause complex failures later on even if they aren't apparent from the desktop.

https://bugzilla.redhat.com/show_bug.cgi?id=695588
 - bluetooth not being started

https://bugzilla.redhat.com/show_bug.cgi?id=696278
 - NetworkManager not being started!

These were just the ones that were obvious because something wasn't working... what other problems are lurking out there?
Comment 14 Tomas Heinrich 2011-05-05 17:26:49 EDT
*** Bug 702425 has been marked as a duplicate of this bug. ***
Comment 15 Adam Williamson 2011-05-06 14:30:53 EDT
could this be caused by https://bugzilla.redhat.com/show_bug.cgi?id=702003 ?

chkconfig --level returning incorrect info for services which have both sysv and systemd initscripts...



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 16 Adam Williamson 2011-05-06 14:31:26 EDT
the triggerun in question uses chkconfig --level , realized i left that out.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 17 Tim Flink 2011-05-06 14:34:14 EDT
Discussed in the 2011-05-06 blocker review meeting. Rejected as F15 final
blocker as it does not clearly hit any release criteria and there is a
reasonable workaround that can be clearly documented. Accepted as NTH because
it would be nice to have this work automatically.
Comment 18 Michal Schmidt 2011-05-06 17:52:42 EDT
There are multiple %triggerun scriptlets in rsyslog. Looks like this is exposing an bug in RPM where only the first trigger is run: bug 702378.
Comment 19 Michal Schmidt 2011-05-06 18:12:22 EDT
(In reply to comment #15)
> could this be caused by https://bugzilla.redhat.com/show_bug.cgi?id=702003 ?
> chkconfig --level returning incorrect info for services which have both sysv
> and systemd initscripts...

No. The %triggerun scriptlet wants to know the information about the old SysV service and that's exactly what chkconfig --level returns.
Comment 20 Jan Vlug 2011-05-10 14:55:42 EDT
I have this bug as well. If I can help with any info let me know.
Comment 21 James Laska 2011-05-16 15:34:50 EDT
theinric: Any updates on this issue ... this is an approved NTH fix for F15, and would alleviate issues if a fix was available in F15 'updates'
Comment 22 Fedora Update System 2011-05-16 15:50:13 EDT
rsyslog-5.7.9-3.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/rsyslog-5.7.9-3.fc15
Comment 23 Adam Williamson 2011-05-16 16:05:52 EDT
note to self/jlaska for commonbugs: remember to explain that you'll have to do the upgrade against a repo with a fixed rsyslog build, to avoid this bug; upgrading to a broken build and then to a fixed one would leave you in broken state, I believe, you'd still have to do the manual fix.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 24 Athmane Madjoudj 2011-05-16 16:18:01 EDT
Seems to be fixed, rsyslog is working again.

May 14 03:04:40 f14-preupgrade NET[1590]: /sbin/dhclient-script : updated /etc/resolv.conf
May 14 03:04:40 f14-preupgrade kernel: Kernel logging (proc) stopped.
May 14 03:04:40 f14-preupgrade rsyslogd: [origin software="rsyslogd" swVersion="4.6.3" x-pid="1032" x-info="http://www.rsyslog.com"] exiting on signal 15.
May 16 20:13:11 f14-preupgrade kernel: imklog 5.7.9, log source = /proc/kmsg started.
May 16 20:13:11 f14-preupgrade rsyslogd: [origin software="rsyslogd" swVersion="5.7.9" x-pid="555" x-info="http://www.rsyslog.com"] start
May 16 20:13:11 f14-preupgrade kernel: [    0.000000] Initializing cgroup subsys cpuset
May 16 20:13:11 f14-preupgrade kernel: [    0.000000] Initializing cgroup subsys cpu
May 16 20:13:11 f14-preupgrade kernel: [    0.000000] Linux version 2.6.38.5-24.fc15.i686 (mockbuild@x86-10.phx2.fedoraproject.org) (gcc version 4.6.0 20110428 (Red Hat 4.6.0-6) (GCC) ) #1 SMP Fri May 6 08:02:58 UTC 2011
[...SNIP...]
May 16 20:13:13 f14-preupgrade audispd: No plugins found, exiting
May 16 20:13:14 f14-preupgrade auditd[869]: Init complete, auditd 2.1.1 listening for events (startup state enable)
May 16 20:13:14 f14-preupgrade systemd[1]: Startup finished in 959ms 885us (kernel) + 1s 525ms 366us (initrd) + 5s 492ms 288us (userspace) = 7s 977ms 539us.
Comment 25 Athmane Madjoudj 2011-05-16 20:50:26 EDT
I've retested with a clean F14 to F15 update with a custom repo from rsyslog-5.7.9-3.fc15 build, injected before starting Anaconda: 

RPMS: http://kojipkgs.fedoraproject.org/packages/rsyslog/5.7.9/3.fc15/i686/

Alternately it possible to use James's repo:

http://jlaska.fedorapeople.org/repos/dist-f15-updates-candidate.repo
Comment 26 Fedora Update System 2011-05-17 01:36:07 EDT
Package rsyslog-5.7.9-3.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing rsyslog-5.7.9-3.fc15'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/rsyslog-5.7.9-3.fc15
then log in and leave karma (feedback).
Comment 27 Tomas Heinrich 2011-05-17 08:33:50 EDT
(In reply to comment #21)
The updated package looks OK. The solution is not perfect but should work reasonably well.
Comment 28 James Laska 2011-05-17 15:55:50 EDT
(In reply to comment #25)
> I've retested with a clean F14 to F15 update with a custom repo from
> rsyslog-5.7.9-3.fc15 build, injected before starting Anaconda: 
> 
> RPMS: http://kojipkgs.fedoraproject.org/packages/rsyslog/5.7.9/3.fc15/i686/
> 
> Alternately it possible to use James's repo:
> 
> http://jlaska.fedorapeople.org/repos/dist-f15-updates-candidate.repo

Thanks!  I also tested and confirmed the fix when the update is included in the anaconda upgrade transation.  Moving to VERIFIED.
Comment 29 Fedora Update System 2011-05-19 01:09:02 EDT
rsyslog-5.7.9-3.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 30 Leif Gruenwoldt 2011-05-28 15:40:12 EDT
Don't think this update fixed the problem. Despite alreading having gotten the update on May 19 and rebooted many a few times since my /var/log/messages is empty.

May 19 20:07:14 Updated: rsyslog-5.7.9-3.fc15.x86_64

Running "$ service rsyslog start" caused messages to immediately fill up.
Comment 31 Adam Williamson 2011-05-29 19:16:18 EDT
it won't fix it if you upgraded to the broken version first, no. we can't really fix that scenario. but it does fix it for people who didn't yet upgrade - if you do an f14-f15 upgrade with the fixed package, it works.
Comment 32 Daniel 2011-10-03 08:24:28 EDT
is it possible to provide updated iso images for the install DVDs?

upgrading via DVD is the "Recommended upgrade method"
http://fedoraproject.org/wiki/Upgrading
but this will lead to the deactivated syslogger without any remarks pointing to the problem or its solution and the logger will not be re-enabled after an update to the fixed package.

i.e. following the recommended upgrade procedure users will end up without a running syslogger and will only learn about this problem when they find empty log files instead of helpful details when they try to debug some issues...
Comment 33 Adam Williamson 2011-10-03 23:54:32 EDT
"is it possible to provide updated iso images for the install DVDs?"

no, not really. we don't do respins except in extremely exceptional (like, eat-your-hardware exceptional) circumstances.
Comment 34 Daniel 2011-10-04 05:27:04 EDT
I see.

But how about spreading some more information about the issue, to help people avoid troubles?

Id suggest to add this bug to the list at http://fedoraproject.org/wiki/Common_F15_bugs#upgrade-disabled-services
I'd also like to see a warning/note on http://fedoraproject.org/wiki/Upgrading which points to http://fedoraproject.org/wiki/Common_F15_bugs#Issues_when_upgrading_from_previous_releases

A remark in the official documentation would also be nice. Possible chapters could be http://docs.fedoraproject.org/en-US/Fedora/15/html/Installation_Guide/ch-upgrade-x86.html or http://docs.fedoraproject.org/en-US/Fedora/15/html/Installation_Guide/sn-finishing-upgrade.html or both.

I understand the reasons for not providing new install media, but not having a system logger running can come back and bite you, so I urge you to spread the informations about this problems to help people prevent empty log files...
Comment 35 Tomas Heinrich 2012-12-04 08:55:47 EST
*** Bug 769918 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.