Red Hat Bugzilla – Bug 601337
debugging of daemons (e.g. NetworkManager) impossible due to missing syslog messages on s390x
Last modified: 2011-05-19 08:29:44 EDT
Description of problem:
Anaconda on s390x does not start a syslogd collecting and writing syslog messages from daemons such as NetworkManager (NM). The output of such daemons would only be visible with a syslogd running.
Most installations on s390x are network installations relying on NM to correctly manage the one network device used for retrieving stage2 (install.img) as well as all the meta data and RPM packages from the yum repository.
We have had at least the following bugs asking for /tmp/syslog on s390x
and it is hardly possible to debug some of those without syslog messages:
A patch backporting the most necessary part of upstream support for rsyslogd for s390x to RHEL6 was rejected
An idea of breaking out the poor man's syslogd implementation of init.c was also rejected in bug 578428 comment 14. Otherwise, s390 could have a standalone minimum syslogd based on that code, just the way it uses a standalone shutdown based on code broken out of init.c/loader.c.
Specifying syslog= as boot option
or logger as kickstart command
is only to redirect syslog messages over the network to a remote syslogd but as far as I understood the code, this even requires a local rsyslogd to be running in the installation image:
"Anaconda supports remote logging handled through the rsyslog daemon running on the installed system. It can be configured to forward its logs through TCP to an arbitrary machine in network that is also running a syslog daemon. This is controlled with the syslog command line option."
Even when specifying "syslog=" as boot option to force loader to pass "--syslog" to anaconda, I do *not* see a syslog start up when anaconda starts from stage2 (which does contain syslogd implementations in its file system).
Even if it did startup at this point during installation, it would be too late for some debugging such as problems in retrieving stage2, since loader already starts NetworkManager much earlier.
Hence, no syslogd gets started and currently there seems no outlook on how to provide one on s390x.
Version-Release number of selected component (if applicable):
Any arbitrary installation on s390x will do.
There is no /tmp/syslog or other place where syslog messages of e.g. NetworkManager appear, not even on any terminal.
/tmp/syslog exists and contains all syslog messages.
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release. Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release. This request is not yet committed for
David, I've pushed the rsyslogd parts:
the daemon should be running in anaconda-13.21.87-1.
(In reply to comment #4)
> David, I've pushed the rsyslogd parts:
> the daemon should be running in anaconda-13.21.87-1.
Unless, I've missed something, the linuxrc.s390 part of
still seems missing, since currently only init.c starts rsyslogd but init.c is replaced by linuxrc.s390 on s390.
(In reply to comment #5)
> (In reply to comment #4)
> > David, I've pushed the rsyslogd parts:
> > 6850919a1bf80dd8d1f6faacd939eafe6a2f3556
> > 3c9de2230c304b48e5f6a658f2cb87482607decb
> > the daemon should be running in anaconda-13.21.87-1.
> Unless, I've missed something, the linuxrc.s390 part of
> still seems missing, since currently only init.c starts rsyslogd but init.c is
> replaced by linuxrc.s390 on s390.
You didn't miss anything. Ales and I discussed the work necessary for this bug and decided he would work in the main rsyslog changes and let me know when those were done, and I'd follow up with the s390 patch(es). So you are responding to a comment he has used to indicate to me that he's done with his work and I can proceed with the s390 work.
/tmp/syslog is created during the installation and contains messages from kernel, NetworkManager, dbus and so on. Messages are also displayed on terminal.
[anaconda root@rtt7 root]# ls -l /tmp
-rw-r--r--. 1 root root 359 Apr 6 12:07 anaconda-yum.conf
-rw-r--r--. 1 root root 5221 Apr 6 12:07 anaconda.log
-rw-r--r--. 1 root root 1215 Apr 6 12:05 e40.conf
-rw-r--r--. 1 root root 3078 Apr 6 12:07 ifcfg.log
-rw-r--r--. 1 root root 125771776 Apr 6 12:06 install.img
-rw-------. 1 root root 235 Apr 6 12:06 libuser.216OsL
-rw-r--r--. 1 root root 16 Apr 6 12:05 modprobe.conf
drwx------. 2 root root 0 Apr 6 12:07 orbit-root
drwxr-xr-x. 5 root root 0 Apr 6 12:06 product
drwxr-xr-x. 2 root root 0 Apr 6 12:06 product-disk
-rw-r--r--. 1 root root 1120 Apr 6 12:07 program.log
-rw-r--r--. 1 root root 5 Apr 6 12:05 s390net
-rw-r--r--. 1 root root 2224 Apr 6 12:06 storage.log
-rw-r--r--. 1 root root 24051 Apr 6 12:06 syslog
drwxr-xr-x. 2 root root 0 Apr 6 12:06 updates
-rwxr-xr-x. 1 root root 576 Apr 6 12:07 vncserver.log
[anaconda root@rtt7 root]# cat /tmp/syslog
12:06:01,905 INFO NetworkManager: <info> (eth0): found s390 'qeth' subchannels [0.0.0a00,0.0.0a01,0.0.0a02]
12:06:01,905 INFO NetworkManager: <info> (eth0): carrier is ON
12:06:01,905 INFO NetworkManager: <info> (eth0): new Ethernet device (driver: 'qeth' ifindex: 2)
12:06:01,905 INFO NetworkManager: <info> (eth0): exported as /org/freedesktop/NetworkManager/Devices/0
12:06:01,905 INFO NetworkManager: <info> (eth0): now managed
12:06:01,905 INFO NetworkManager: <info> (eth0): device state change: 1 -> 2 (reason 41)
12:06:01,905 INFO NetworkManager: <info> (eth0): preparing device.
12:06:01,906 INFO NetworkManager: <info> Activation (eth0) starting connection 'System eth0'
Tested on build RHEL6.1-20110330.2 with anaconda-13.21.108-1.el6.
Moving to VERIFIED
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.