When commands run by systemd fail, their stdout and stderr seem to go to /dev/null instead of to boot.log and syslog as they should. This makes it incredibly difficult to track down problems. I filed a bug report about systemd failing to successfully mount an NFS filesystem, but I am cloning the bug here to make sure that the issue of lost errors is recognized as a problem in and of itself. By the way, I'm sure I've run into this with errors from other commands run by systemd, too (and not just from mount). +++ This bug was initially created as a clone of Bug #749940 +++ Description of problem: I have an NFS mount in /etc/fstab called /aml. On my Fedora 16 Beta machine, the systemd unit for this mount fails at startup, and I can't find any indication of what's wrong. In /var/log/boot.log, I see the following: Starting /aml ESC[1;31mfailedESC[0m, see 'systemctl status aml.mount' for details. If I run "systemctl status aml.mount", it gives: aml.mount - /aml Loaded: loaded Active: failed since Fri, 28 Oct 2011 16:42:45 -0600; 11min ago Where: /aml What: nfs:/ Process: 372 ExecMount=/bin/mount /aml (code=exited, status=32) CGroup: name=systemd:/system/aml.mount Unfortunately, I can't find any logs that show what the specific error message is. Additionally, if I mount the filesystem manually with "mount /aml", it succeeds. Version-Release number of selected component (if applicable): systemd-37-2.fc16.x86_64 How reproducible: every boot Steps to Reproduce: 1. Add an NFS mount to /etc/fstab. 2. Boot. Actual results: The mount unit fails. Expected results: The filesystem should be mounted. Additional info: Because of this problem, NFS home directories are unavailable without manual intervention. If it helps, the specific line in fstab is: nfs:/ /aml nfs4 noatime,intr,actimeo=0 0 0
Hmm, what does the following command output for you? systemctl show aml.mount | grep ^Standard On F16 all units should log to syslog by default.
$ systemctl show aml.mount | grep ^Standard StandardInput=null StandardOutput=kmsg StandardError=inherit $ dmesg |grep -i aml [ 2.540824] systemd[1]: Set hostname to <testvm.aml.cs.byu.edu>. [ 3.005320] systemd[1]: aml.mount mount process exited, code=exited status=32 [ 3.030421] systemd[1]: Unit aml.mount entered failed state. $ grep aml /var/log/messages $
I'm also noticing the rc-local.service is failing without any indication in the logs of what could be going on. Interestingly, its standard output is not the same as the above: $ systemctl show rc-local.service |grep ^Standard StandardInput=null StandardOutput=tty StandardError=inherit $ grep rc-local /var/log/messages Nov 1 14:11:02 sage systemd[1]: rc-local.service: control process exited, code=exited status=203 Nov 1 14:11:02 sage systemd[1]: Unit rc-local.service entered failed state. $ Is there anything else you would like me to try? In both Fedora 15 and Fedora 16, I've found that it's very difficult to track down why a service fails to start.
The rc-local.service unit file asks for it explicitly: StandardOutput=tty We should drop that. It should write to syslog like other services.
On Fedora 16 I had a similar issue when adding the ethtool command to rc.local. The service would report failure error 203. The end result, I forgot to add the #!/bin/sh interpreter line to the top of the /etc/rc.d/rc.local script. Adding the interpreter fixed the 203 error. Hope this helps since it could be an easy mistake. Duplicate the issue: 1. Create the /etc/rc.d/rc.local file and chmod +x with the following two ethtool commands /usr/sbin/ethtool -s p1p1 speed 100 duplex full autoneg off /usr/sbin/ethtool -s em1 speed 100 duplex full autoneg off 2. Test the script by restarting the rc-local.service $ sudo systemctl restart rc-local.service Job failed. See system logs and 'systemctl status' for details. 3. Check the status $ systemctl status rc-local.service rc-local.service - /etc/rc.local Compatibility Loaded: loaded (/lib/systemd/system/rc-local.service; linked) Active: failed since Fri, 11 Nov 2011 11:28:44 -0600; 16s ago Process: 6312 ExecStart=/etc/rc.d/rc.local start (code=exited, status=203/EXEC) CGroup: name=systemd:/system/rc-local.service 4. Now, fix the rc.local file as follows: #!/bin/sh /usr/sbin/ethtool -s p1p1 speed 100 duplex full autoneg off /usr/sbin/ethtool -s em1 speed 100 duplex full autoneg off 5. Restart and status $ sudo systemctl restart rc-local.service $ systemctl status rc-local.service rc-local.service - /etc/rc.local Compatibility Loaded: loaded (/lib/systemd/system/rc-local.service; linked) Active: active (exited) since Fri, 11 Nov 2011 11:30:19 -0600; 32s ago Process: 6541 ExecStart=/etc/rc.d/rc.local start (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/rc-local.service
(In reply to comment #5) > On Fedora 16 I had a similar issue when adding the ethtool command to rc.local. > The service would report failure error 203. > > The end result, I forgot to add the #!/bin/sh interpreter line to the top of > the /etc/rc.d/rc.local script. Adding the interpreter fixed the 203 error. > > Hope this helps since it could be an easy mistake. This turns out to be the error with the rc.local script. This doesn't help with the failed mount command, but it's a good example of a place where systemd could give a helpful error message.
(In reply to comment #5) > The end result, I forgot to add the #!/bin/sh interpreter line to the top of > the /etc/rc.d/rc.local script. Adding the interpreter fixed the 203 error. This is being discussed in bug 752901. (In reply to comment #6) > This turns out to be the error with the rc.local script. This doesn't help with > the failed mount command, but it's a good example of a place where systemd > could give a helpful error message. For rc-local.service this is fixed upstream: http://cgit.freedesktop.org/systemd/commit/?id=9d7286112d8b54da0294ca0d6bbd7314d890b7e2 No fix available yet for the mount units.
I see two bugs here: 1) "StandardOutput=kmsg". We should let mount units log to the default destination. I can fix that. 2) On my system the logging of mount commands is prevented by SELinux: bug 754938
Issue 1) fixed upstream: http://cgit.freedesktop.org/systemd/commit/?id=f6cebb3bd5a00d79c8131637c0f6796a75e6af99
systemd-37-6.fc16 has been submitted as an update for Fedora 16. https://admin.fedoraproject.org/updates/systemd-37-6.fc16
Package systemd-37-6.fc16: * should fix your issue, * was pushed to the Fedora 16 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing systemd-37-6.fc16' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-0409/systemd-37-6.fc16 then log in and leave karma (feedback).
Package systemd-37-7.fc16: * should fix your issue, * was pushed to the Fedora 16 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing systemd-37-7.fc16' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-0409/systemd-37-7.fc16 then log in and leave karma (feedback).
Package systemd-37-8.fc16: * should fix your issue, * was pushed to the Fedora 16 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing systemd-37-8.fc16' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-0409/systemd-37-8.fc16 then log in and leave karma (feedback).
Package systemd-37-10.fc16: * should fix your issue, * was pushed to the Fedora 16 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing systemd-37-10.fc16' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-0409/systemd-37-10.fc16 then log in and leave karma (feedback).
Package systemd-37-11.fc16: * should fix your issue, * was pushed to the Fedora 16 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing systemd-37-11.fc16' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-0409/systemd-37-11.fc16 then log in and leave karma (feedback).
systemd-37-11.fc16 has been pushed to the Fedora 16 stable repository. If problems still persist, please make note of it in this bug report.