Bug 699198

I see this as well, in fact, rsyslog has never worked for me since I upgraded to F15. Last messages are the normal shutdown on F14 while running the preupgrade and then nothing.

Version:
rsyslog.x86_64      5.7.9-2.fc15

After boot, I get:

# service rsyslog status
rsyslog.service - System Logging Service
          Loaded: loaded (/lib/systemd/system/rsyslog.service)
          Active: inactive (dead)
          CGroup: name=systemd:/system/rsyslog.service

If I start the service manually it then works just fine (and includes all messages since last boot).

# chkconfig --list | grep syslog

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overriden by native
      systemd configuration.

rsyslog         0:off   1:off   2:on    3:on    4:on    5:on    6:off

I'm not familiar with systemd to understand if rsyslog is even trying to start at boot.

(In reply to comment #0)
Is the rsyslogd process still running?
Do you have selinux in enforcing mode?
Do you have selinux-policy updated to the latest version?
Are there any relevant AVCs in /var/log/audit/audit.log?
  $ ausearch -m avc -c rsyslog
What's the output of
  $ systemctl status rsyslog.service
Does restarting the service help?
Is rsyslog configured to start with systemd?
  $ ls /etc/systemd/system/*.wants/rsyslog.service

(In reply to comment #1)
The same questions here.
Here's some info re systemd troubleshooting:
http://fedoraproject.org/wiki/How_to_debug_Systemd_problems

(In reply to comment #2)
> (In reply to comment #0)
> Is the rsyslogd process still running?
No, even after reboots

> Do you have selinux in enforcing mode?
SELINUX=enforcing

> Do you have selinux-policy updated to the latest version?
yes, regular yum updates and the first thing I did on seeing this problem was to relabel everything, just in case.

> Are there any relevant AVCs in /var/log/audit/audit.log?
>   $ ausearch -m avc -c rsyslog
<no matches>
(interesting command, I'll remember it)

> What's the output of
>   $ systemctl status rsyslog.service
rsyslog.service - System Logging Service
	  Loaded: loaded (/lib/systemd/system/rsyslog.service)
	  Active: inactive (dead)
	  CGroup: name=systemd:/system/rsyslog.service


> Does restarting the service help?
Only until the next boot, it seems

> Is rsyslog configured to start with systemd?
>   $ ls /etc/systemd/system/*.wants/rsyslog.service
no such file, so I suppose no

> Here's some info re systemd troubleshooting:
> http://fedoraproject.org/wiki/How_to_debug_Systemd_problems

It looks like it's failing to start somehow at boot time, I'm not sure where to go from here.

Created attachment 495118 [details]
debug output mentioned in wiki

The symptoms of this bug can probably be easily remedied by running
  # systemctl enable rsyslog.service

The cause seems to be that the rsyslog service is not enabled when upgrading from a version that uses a SysV script (/etc/rc.d/init.d/rsyslog)
to a version that uses a systemd unit file (/lib/systemd/system/rsyslog.service),
even if the service was enabled in chkconfig.

Not sure what's the right way to fix this. Lennart?

(In reply to comment #2)
> (In reply to comment #1)
> The same questions here.
> Here's some info re systemd troubleshooting:
> http://fedoraproject.org/wiki/How_to_debug_Systemd_problems
My results are exactly the same as those in comment #3.

(In reply to comment #5)
> The symptoms of this bug can probably be easily remedied by running
>   # systemctl enable rsyslog.service
Thank you for the workaround, service starts on boot now and the setting persists across reboot cycles.

(In reply to comment #5)
> The symptoms of this bug can probably be easily remedied by running
>   # systemctl enable rsyslog.service
> 
> The cause seems to be that the rsyslog service is not enabled when upgrading
> from a version that uses a SysV script (/etc/rc.d/init.d/rsyslog)
> to a version that uses a systemd unit file
> (/lib/systemd/system/rsyslog.service),
> even if the service was enabled in chkconfig.
> 
> Not sure what's the right way to fix this. Lennart?

Fascinating... ideally we would have some way to detect that something is missing before we come to use it... I'll bet many system services might be missing like this after upgrade. No wonder it boots faster ;-) Is there a mechanical way to audit it?

The code to handle the upgrade is actually existant in the spec file.

http://pkgs.fedoraproject.org/gitweb/?p=rsyslog.git;a=blob;f=rsyslog.spec;h=b5102d3eccf4d942f1c8fc3e8bb433545f2b4dc4;hb=refs/heads/f15/master#l219

If you run that excerpt manually, does it work then for you?

(In reply to comment #8)
> The code to handle the upgrade is actually existant in the spec file.
> 
> http://pkgs.fedoraproject.org/gitweb/?p=rsyslog.git;a=blob;f=rsyslog.spec;h=b5102d3eccf4d942f1c8fc3e8bb433545f2b4dc4;hb=refs/heads/f15/master#l219
> 
> If you run that excerpt manually, does it work then for you?

Sorry, I'm not sure exactly what code to run or if this is now a valid context to run it in - having manually applied the remedy in comment #5. I take it you mean the stuff in the %post, which just seems to do 

 chkconfig --add rsyslog

there is also some conditional stuff in there that says it's for install, not upgrade - I take it in my case it would have been an upgrade. I still have the upgrade.log and upgrade.log.syslog files if that would help... The upgrade log mentions chkconfig and rsyslog in the correct order:

 Upgrading chkconfig-1.3.51-1.fc15.i686
 [...]
 Upgrading rsyslog-5.7.9-2.fc15.i686

(In reply to comment #9)
I think the code in question is this:
 219 %triggerun -- rsyslog < 5.7.8-1
 220 if /sbin/chkconfig --level 3 rsyslog ; then
 221         /bin/systemctl --no-reload enable rsyslog.service >/dev/null 2>&1 || :
 222 fi

For me, running it manually works.

With rpm, I get this:

# systemctl disable rsyslog.service
rm '/etc/systemd/system/multi-user.target.wants/rsyslog.service'
# rpm --force -U rsyslog-4.6.3-2.fc15.x86_64.rpm
# chkconfig --level 2345 rsyslog on
# rpm -vv -U rsyslog-5.7.9-2.fc15.x86_64.rpm
<...>
D:     0    0    0    1   +rsyslog-5.7.9-2.fc15.x86_64
D:     1    0    0    1   -rsyslog-4.6.3-2.fc15.x86_64
<...>
D: %post(rsyslog-5.7.9-2.fc15.x86_64): waitpid(5338) rc 5338 status 0
D:  read h#     317 Header SHA1 digest: OK (bd3fe00e9e67d3d431adeb8882827344b800840a)
D: ========== +++ rsyslog-4.6.3-2.fc15 x86_64-linux 0x2
D:     erase: rsyslog-4.6.3-2.fc15 has 135 files
D: (null): scriptlet start
D: (null): execv(/bin/sh) pid 5343
+ '[' -f /var/lock/subsys/rsyslogd ']'
+ mv /var/lock/subsys/rsyslogd /var/lock/subsys/rsyslog
D: (null): waitpid(5343) rc 5343 status 0
D: %preun(rsyslog-4.6.3-2.fc15.x86_64): scriptlet start
<...>

Looks like only the second 'triggerun' gets run and the third one doesn't.

Having the same issues with no logging but with a twist:
systemctl status rsyslog.service fails with a 
Failed to get D-Bus connection: Failed to connect to socket
/org/freedesktop/systemd1/private: Connection refused

* Systemadm also fails with this error.

I have no dmesg or messages logged.

Chkconfig --add rsyslog.service did not error.

I have not lines in my grub.conf to tell systemd to start and I thought there was meant to be a line!

I have an F14 virtual machine upgraded to F15 Beta1 and then patched upto date using package manager.

I just ran into this bug after updating from F14 to F15.  Is there some way to check for other services that got lost during the update?

Thanks,
Gene

(In reply to comment #12)
> I just ran into this bug after updating from F14 to F15.  Is there some way to
> check for other services that got lost during the update?
> 
> Thanks,
> Gene

I completely agree, an automated method that will root out all of these problems is desperately needed. It seems there are several cases of different services not enabled after upgrade (when they should have been). Some cause complex failures later on even if they aren't apparent from the desktop.

https://bugzilla.redhat.com/show_bug.cgi?id=695588
 - bluetooth not being started

https://bugzilla.redhat.com/show_bug.cgi?id=696278
 - NetworkManager not being started!

These were just the ones that were obvious because something wasn't working... what other problems are lurking out there?

*** Bug 702425 has been marked as a duplicate of this bug. ***

could this be caused by https://bugzilla.redhat.com/show_bug.cgi?id=702003 ?

chkconfig --level returning incorrect info for services which have both sysv and systemd initscripts...



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

the triggerun in question uses chkconfig --level , realized i left that out.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Discussed in the 2011-05-06 blocker review meeting. Rejected as F15 final
blocker as it does not clearly hit any release criteria and there is a
reasonable workaround that can be clearly documented. Accepted as NTH because
it would be nice to have this work automatically.

There are multiple %triggerun scriptlets in rsyslog. Looks like this is exposing an bug in RPM where only the first trigger is run: bug 702378.

(In reply to comment #15)
> could this be caused by https://bugzilla.redhat.com/show_bug.cgi?id=702003 ?
> chkconfig --level returning incorrect info for services which have both sysv
> and systemd initscripts...

No. The %triggerun scriptlet wants to know the information about the old SysV service and that's exactly what chkconfig --level returns.

I have this bug as well. If I can help with any info let me know.

theinric: Any updates on this issue ... this is an approved NTH fix for F15, and would alleviate issues if a fix was available in F15 'updates'

rsyslog-5.7.9-3.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/rsyslog-5.7.9-3.fc15

note to self/jlaska for commonbugs: remember to explain that you'll have to do the upgrade against a repo with a fixed rsyslog build, to avoid this bug; upgrading to a broken build and then to a fixed one would leave you in broken state, I believe, you'd still have to do the manual fix.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Seems to be fixed, rsyslog is working again.

May 14 03:04:40 f14-preupgrade NET[1590]: /sbin/dhclient-script : updated /etc/resolv.conf
May 14 03:04:40 f14-preupgrade kernel: Kernel logging (proc) stopped.
May 14 03:04:40 f14-preupgrade rsyslogd: [origin software="rsyslogd" swVersion="4.6.3" x-pid="1032" x-info="http://www.rsyslog.com"] exiting on signal 15.
May 16 20:13:11 f14-preupgrade kernel: imklog 5.7.9, log source = /proc/kmsg started.
May 16 20:13:11 f14-preupgrade rsyslogd: [origin software="rsyslogd" swVersion="5.7.9" x-pid="555" x-info="http://www.rsyslog.com"] start
May 16 20:13:11 f14-preupgrade kernel: [    0.000000] Initializing cgroup subsys cpuset
May 16 20:13:11 f14-preupgrade kernel: [    0.000000] Initializing cgroup subsys cpu
May 16 20:13:11 f14-preupgrade kernel: [    0.000000] Linux version 2.6.38.5-24.fc15.i686 (mockbuild.fedoraproject.org) (gcc version 4.6.0 20110428 (Red Hat 4.6.0-6) (GCC) ) #1 SMP Fri May 6 08:02:58 UTC 2011
[...SNIP...]
May 16 20:13:13 f14-preupgrade audispd: No plugins found, exiting
May 16 20:13:14 f14-preupgrade auditd[869]: Init complete, auditd 2.1.1 listening for events (startup state enable)
May 16 20:13:14 f14-preupgrade systemd[1]: Startup finished in 959ms 885us (kernel) + 1s 525ms 366us (initrd) + 5s 492ms 288us (userspace) = 7s 977ms 539us.

I've retested with a clean F14 to F15 update with a custom repo from rsyslog-5.7.9-3.fc15 build, injected before starting Anaconda: 

RPMS: http://kojipkgs.fedoraproject.org/packages/rsyslog/5.7.9/3.fc15/i686/

Alternately it possible to use James's repo:

http://jlaska.fedorapeople.org/repos/dist-f15-updates-candidate.repo

Package rsyslog-5.7.9-3.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing rsyslog-5.7.9-3.fc15'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/rsyslog-5.7.9-3.fc15
then log in and leave karma (feedback).

(In reply to comment #21)
The updated package looks OK. The solution is not perfect but should work reasonably well.

(In reply to comment #25)
> I've retested with a clean F14 to F15 update with a custom repo from
> rsyslog-5.7.9-3.fc15 build, injected before starting Anaconda: 
> 
> RPMS: http://kojipkgs.fedoraproject.org/packages/rsyslog/5.7.9/3.fc15/i686/
> 
> Alternately it possible to use James's repo:
> 
> http://jlaska.fedorapeople.org/repos/dist-f15-updates-candidate.repo

Thanks!  I also tested and confirmed the fix when the update is included in the anaconda upgrade transation.  Moving to VERIFIED.

rsyslog-5.7.9-3.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

Don't think this update fixed the problem. Despite alreading having gotten the update on May 19 and rebooted many a few times since my /var/log/messages is empty.

May 19 20:07:14 Updated: rsyslog-5.7.9-3.fc15.x86_64

Running "$ service rsyslog start" caused messages to immediately fill up.

it won't fix it if you upgraded to the broken version first, no. we can't really fix that scenario. but it does fix it for people who didn't yet upgrade - if you do an f14-f15 upgrade with the fixed package, it works.

is it possible to provide updated iso images for the install DVDs?

upgrading via DVD is the "Recommended upgrade method"
http://fedoraproject.org/wiki/Upgrading
but this will lead to the deactivated syslogger without any remarks pointing to the problem or its solution and the logger will not be re-enabled after an update to the fixed package.

i.e. following the recommended upgrade procedure users will end up without a running syslogger and will only learn about this problem when they find empty log files instead of helpful details when they try to debug some issues...

"is it possible to provide updated iso images for the install DVDs?"

no, not really. we don't do respins except in extremely exceptional (like, eat-your-hardware exceptional) circumstances.

I see.

But how about spreading some more information about the issue, to help people avoid troubles?

Id suggest to add this bug to the list at http://fedoraproject.org/wiki/Common_F15_bugs#upgrade-disabled-services
I'd also like to see a warning/note on http://fedoraproject.org/wiki/Upgrading which points to http://fedoraproject.org/wiki/Common_F15_bugs#Issues_when_upgrading_from_previous_releases

A remark in the official documentation would also be nice. Possible chapters could be http://docs.fedoraproject.org/en-US/Fedora/15/html/Installation_Guide/ch-upgrade-x86.html or http://docs.fedoraproject.org/en-US/Fedora/15/html/Installation_Guide/sn-finishing-upgrade.html or both.

I understand the reasons for not providing new install media, but not having a system logger running can come back and bite you, so I urge you to spread the informations about this problems to help people prevent empty log files...

*** Bug 769918 has been marked as a duplicate of this bug. ***