Bug 633774
| Summary: | systemd doesn't bring up sendmail even though sysvinit links exist | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Mirco Tischler <mt-ml> | ||||
| Component: | systemd | Assignee: | Lennart Poettering <lpoetter> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 15 | CC: | ADent123, bugs.michael, Colin.Simpson, craig, dwm, igeorgex, john, kmassare, lpoetter, maurizio.antillon, mcepl, metherid, mschmidt, mt-ml, notting, orion, pbrobinson, pcfe, plautrba, umar, vonbrand, warlord | ||||
| Target Milestone: | --- | Keywords: | Reopened, Triaged | ||||
| Target Release: | --- | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | systemd-26-8.fc15 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2011-07-15 01:23:07 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 494832 | ||||||
| Attachments: |
|
||||||
Something attempts to reload sendmail while it is scheduled for start. init[1]: Added job sendmail.service/start to transaction. ... init[1]: Accepted connection on private bus. init[1]: Got D-Bus request: org.freedesktop.systemd1.Manager.ReloadUnit() on /org/freedesktop/systemd1 init[1]: Trying to enqueue job sendmail.service/reload/replace init[1]: Added job sendmail.service/reload to transaction. init[1]: Enqueued job sendmail.service/reload as 197 init[1]: Got D-Bus request: org.freedesktop.systemd1.Manager.GetUnit() on /org/freedesktop/systemd1 init[1]: Got D-Bus request: org.freedesktop.DBus.Properties.Get() on /org/freedesktop/systemd1/unit/sendmail_2eservice Is it /etc/NetworkManager/dispatcher.d/10-sendmail? Does it help if move this file away temporarily? Yes. moving this file somewhere else does allow sendmail to start. Moving systemd bugs to f15, since the systemd feature got delayed. *** Bug 665888 has been marked as a duplicate of this bug. *** Marked as triaged for duplicates, and because there doesn't seem to be anything to clarify here. This works now... (In reply to comment #6) > This works now... Sorry, no. It seems that it works sometimes, mostly it doesn't. Could you please try to reproduce this and when you manage to do a "systemctl show sendmail.service" and paste the output here? It seems to have dissapeared, hasn't happened again for at least a week or so. systemd-18-1.fc16.x86_64 sendmail-8.14.4-20.fc15.x86_64 OK, closing then. Feel free to reopen if the problem appears again. I can reproduce on my VM.
systemd-24-1.fc15.x86_64
sendmail-8.14.4-20.fc15.x86_64
# systemctl show sendmail.service
Id=sendmail.service
Names=sendmail.service
Requires=basic.target
Wants=mail-transfer-agent.target
WantedBy=multi-user.target graphical.target
Conflicts=shutdown.target
Before=mail-transfer-agent.target puppet.service shutdown.target smolt.service multi-user.target
After=local-fs.target network.target basic.target sandbox.service netconsole.service auditd.serv
Description=LSB: start and stop sendmail
LoadState=loaded
ActiveState=inactive
SubState=dead
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
DefaultControlGroup=name=systemd:/system/sendmail.service
ControlGroup=cpu:/system/sendmail.service name=systemd:/system/sendmail.service
NeedDaemonReload=no
JobTimeoutUSec=0
ConditionResult=no
Type=forking
Restart=no
PIDFile=/var/run/sendmail.pid
NotifyAccess=none
RestartUSec=100ms
TimeoutUSec=5min
ExecStart={ path=/etc/rc.d/init.d/sendmail ; argv[]=/etc/rc.d/init.d/sendmail start ; ignore=no
ExecReload={ path=/etc/rc.d/init.d/sendmail ; argv[]=/etc/rc.d/init.d/sendmail reload ; ignore=n
ExecStop={ path=/etc/rc.d/init.d/sendmail ; argv[]=/etc/rc.d/init.d/sendmail stop ; ignore=no ;
UMask=0002
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=1024
LimitAS=18446744073709551615
LimitNPROC=5814
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=5814
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=inherit
StandardError=inherit
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSet=18446744073709551615
MountFlags=1048576
PrivateTmp=no
SameProcessGroup=no
KillMode=process
KillSignal=15
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=yes
GuessMainPID=yes
ExecMainPID=0
ExecMainCode=0
ExecMainStatus=0
MainPID=0
ControlPID=0
SysVRunLevels=2345
SysVStartPriority=80
SysVPath=/etc/rc.d/init.d/sendmail
FsckPassNo=0
# dmesg | grep -F mail [ 4.910838] systemd[1]: Installed new job sendmail.service/start as 124 [ 4.910845] systemd[1]: Installed new job mail-transfer-agent.target/start as 125 [ 17.220081] systemd[1]: Trying to enqueue job sendmail.service/reload/replace [ 17.220132] systemd[1]: Installed new job sendmail.service/reload as 229 [ 17.220138] systemd[1]: Enqueued job sendmail.service/reload as 229 [ 17.220514] systemd[1]: Got D-Bus request: org.freedesktop.DBus.Properties.Get() on /org/freedesktop/systemd1/unit/sendmail_2eservice [ 32.618259] systemd[1]: Job sendmail.service/reload finished, result=skipped [ 32.618411] systemd[1]: mail-transfer-agent.target changed dead -> active [ 32.618670] systemd[1]: Job mail-transfer-agent.target/start finished, result=done [ 99.121768] systemd[1]: Got D-Bus request: org.freedesktop.DBus.Properties.GetAll() on /org/freedesktop/systemd1/unit/sendmail_2eservice [ 171.819208] systemd[1]: Got D-Bus request: org.freedesktop.DBus.Properties.GetAll() on /org/freedesktop/systemd1/unit/sendmail_2eservice [ 319.441741] systemd[1]: Trying to enqueue job sendmail.service/start/replace [ 319.442353] systemd[1]: Installed new job sendmail.service/start as 235 [ 319.442359] systemd[1]: Enqueued job sendmail.service/start as 235 [ 319.442446] systemd[1]: About to execute: /etc/rc.d/init.d/sendmail start [ 319.452781] systemd[1]: Forked /etc/rc.d/init.d/sendmail as 1085 [ 319.458004] systemd[1]: sendmail.service changed dead -> start [ 319.463707] systemd[1]: Got D-Bus request: org.freedesktop.DBus.Properties.Get() on /org/freedesktop/systemd1/unit/sendmail_2eservice [ 320.169119] systemd[1]: Received SIGCHLD from PID 1085 (sendmail). [ 320.172744] systemd[1]: Got SIGCHLD for process 1085 (sendmail) [ 320.180340] systemd[1]: Child 1085 belongs to sendmail.service [ 320.180353] systemd[1]: sendmail.service: control process exited, code=exited status=0 [ 320.180358] systemd[1]: sendmail.service got final SIGCHLD for state start [ 320.181772] systemd[1]: sendmail.service changed start -> running [ 320.181787] systemd[1]: Job sendmail.service/start finished, result=done [ 320.446863] systemd[1]: Received SIGCHLD from PID 1111 (sendmail). [ 320.447311] systemd[1]: Got SIGCHLD for process 1111 (sendmail) Last (319-320) is where is started by hand with service sendmail start. Okay, some lines were trimmed:
Before=mail-transfer-agent.target puppet.service shutdown.target smolt.service multi-user.target graphical.target
After=local-fs.target network.target basic.target sandbox.service netconsole.service auditd.service rpcbind.service mysqld.service speech-dispatcherd.service yum-cron.service
ExecStart={ path=/etc/rc.d/init.d/sendmail ; argv[]=/etc/rc.d/init.d/sendmail start ; ignore=no ; start_time=[n/a] ; stop_time=[Mon, 11 Apr 2011 16:05:40 -0600] ; pid=1085 ; code=exited ; status=0 }
ExecReload={ path=/etc/rc.d/init.d/sendmail ; argv[]=/etc/rc.d/init.d/sendmail reload ; ignore=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/ }
ExecStop={ path=/etc/rc.d/init.d/sendmail ; argv[]=/etc/rc.d/init.d/sendmail stop ; ignore=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/ }
I can reproduce this with a little help to expose the race condition. Enabling a unit like this does it nicely: [Unit] Description=MMM delay sendmail for bz633774 Before=sendmail.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/bin/sleep 15 [Install] WantedBy=multi-user.target The race is: 1. The job "sendmail.service/start" is installed. 2. Before systemd proceeds with this job, something (see comment #1) asks for reload of sendmail. The job "sendmail.service/reload/replace" is enqueued. 3. The original job is forgotten (because of the replace mode of the new one?) I have this under F15. Is this bug tracking that too? > Is it /etc/NetworkManager/dispatcher.d/10-sendmail? Does it help if move this > file away temporarily? Looks like that is a problem with the NM dispatcher scripts that were documented in bug 703321 and should now be fixed. Please confirm. FYI, i just upgraded two systems to F15 over the last couple days and I'm seeing this problem with both sendmail and squid. Both have files in /etc/NetworkManager/dispatcher.d. But no, this file still exists as of yesterday (I didn't freshen the update today). I can also verify that removing the offending service file from the NM dispatcher makes the problem go away. The system is fully updated as of minutes ago. (In reply to comment #16) > Looks like that is a problem with the NM dispatcher scripts that were > documented in bug 703321 and should now be fixed. Please confirm. No, the breakage of NM dispatcher scripts did not cause this bug. The race I described in comment #14 exposes a real bug in systemd. Just hit this bug.....I did not have the problem until yesterday since I did not have NetworkManager enabled until then. After enabling and rebooting sendmail did not start. Not running squid here, but am not seeing sendmail running after boot. Manual starts are fine. This build should fix it. Please give it a try: http://koji.fedoraproject.org/koji/taskinfo?taskID=3175735 Pushed the patch upstream: http://cgit.freedesktop.org/systemd/commit/?id=1b562e4604f8833bc21fd251b8bdb45c9c224df4 (In reply to comment #22) > This build should fix it. Please give it a try: > http://koji.fedoraproject.org/koji/taskinfo?taskID=3175735 Works for me. Thanks. systemd-26-6.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/systemd-26-6.fc15 Package systemd-26-7.fc15: * should fix your issue, * was pushed to the Fedora 15 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing systemd-26-7.fc15' as soon as you are able to, then reboot. Please go to the following url: https://admin.fedoraproject.org/updates/systemd-26-7.fc15 then log in and leave karma (feedback). Since updating to the systemd-26-7.fc15 packages, sendmail fails to start here at x86_64 boot time. Downgrading to the -5.fc15 packages fixes it. Installed systemd-26-7.fc15 on an x86 system. The system booted to Emergency maintenance prompt. Indications were that fsck.ext3 failed to read superblock on /dev/md0. Restored systemd back to 26-5 and all was well (except sendmail did not load on boot). I edited /etc/rc.d/rc.local by adding a line /sbin/service sendmail start as a temporary workaround. I have another server also running the x86 version of Fedora 15 on which the sendmail service starts on boot up. This box is not running spamassassin unlike the server that fails. Installed systemd-26-7.fc15 on an x86 system. The system booted to Emergency maintenance prompt. Indications were that fsck.ext3 failed to read superblock on /dev/md0. Restored systemd back to 26-5 and all was well (except sendmail did not load on boot). I edited /etc/rc.d/rc.local by adding a line /sbin/service sendmail start as a temporary workaround. I have another server also running the x86 version of Fedora 15 on which the sendmail service starts on boot up. This box is not running spamassassin unlike the server that fails. Package systemd-26-8.fc15: * should fix your issue, * was pushed to the Fedora 15 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing systemd-26-8.fc15' as soon as you are able to, then reboot. Please go to the following url: https://admin.fedoraproject.org/updates/systemd-26-8.fc15 then log in and leave karma (feedback). (In reply to comment #28) > Installed systemd-26-7.fc15 on an x86 system. The system booted to Emergency > maintenance prompt. Indications were that fsck.ext3 failed to read superblock > on /dev/md0 Does it happen with systemd-26-8.fc15 too? Updated to systemd-26-8.fc15 and had the same issue with /dev/md0 superblock. Here is the end of dmesg with the failure:
[ 18.149197] md: bind<sdb1>
[ 18.630044] systemd-fsck[709]: fsck.ext3: Invalid argument while trying to open /dev/md0
[ 18.630055] systemd-fsck[709]: /dev/md0:
[ 18.630064] systemd-fsck[709]: The superblock could not be read or does not describe a correct ext2
[ 18.630073] systemd-fsck[709]: fsck failed with error code 8.
[ 18.630082] systemd-fsck[709]: filesystem. If the device is valid and it really contains an ext2
[ 18.630091] systemd-fsck[709]: Ignoring error.
[ 18.630100] systemd-fsck[709]: filesystem (and not swap or ufs or something else), then the superblock
[ 18.630109] systemd-fsck[709]: is corrupt, and you might try running e2fsck with an alternate superblock:
[ 18.630117] systemd-fsck[709]: e2fsck -b 8193 <device>
[ 18.651959] EXT3-fs (md0): error: unable to read superblock
[ 18.655247] mount[715]: mount: wrong fs type, bad option, bad superblock on /dev/md0,
[ 18.655255] mount[715]: missing codepage or helper program, or other error
[ 18.655274] mount[715]: (could this be the IDE device where you in fact use
[ 18.655280] mount[715]: ide-scsi so that sr0 or sda or so is needed?)
[ 18.655290] mount[715]: In some cases useful info is found in syslog - try
[ 18.655295] mount[715]: dmesg | tail or so
[ 18.655603] systemd[1]: var.mount mount process exited, code=exited status=32
[ 18.664118] systemd[1]: Job var-lock.mount/start failed with result 'dependency'.
[ 18.664202] systemd[1]: Job avahi-daemon.service/start failed with result 'dependency'.
[ 18.664211] systemd[1]: Job avahi-daemon.socket/start failed with result 'dependency'.
[ 18.664262] systemd[1]: Job dbus.service/start failed with result 'dependency'.
[ 18.664307] systemd[1]: Job NetworkManager.service/start failed with result 'dependency'.
[ 18.664352] systemd[1]: Job abrtd.service/start failed with result 'dependency'.
[ 18.664360] systemd[1]: Job dbus.socket/start failed with result 'dependency'.
[ 18.664369] systemd[1]: Job var-run.mount/start failed with result 'dependency'.
[ 18.664417] systemd[1]: Job fedora-autorelabel.service/start failed with result 'dependency'.
[ 18.664462] systemd[1]: Job fedora-autorelabel-mark.service/start failed with result 'dependency'.
[ 18.664471] systemd[1]: Job local-fs.target/start failed with result 'dependency'.
[ 18.664479] systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
[ 18.665995] systemd[1]: Unit var.mount entered failed state.
[ 18.682060] md: bind<sda5>
[ 19.887134] systemd[1]: Startup finished in 2s 586ms 698us (kernel) + 3s 539ms 27us (initrd) + 13s 761ms 332us (userspace) = 19s 887ms 57us.
[ 34.173117] kvm: disable TXT in the BIOS or activate TXT before enabling KVM
[ 34.173120] kvm: disabled by bios
[ 34.338736] udevd[764]: specified group 'cdrom' unknown
[ 34.339346] systemd-fsck[750]: fsck.ext3: Invalid argument while trying to open /dev/md0
[ 34.339640] systemd-fsck[750]: /dev/md0:
[ 34.339821] systemd-fsck[750]: The superblock could not be read or does not describe a correct ext2
[ 34.340110] systemd-fsck[750]: filesystem. If the device is valid and it really contains an ext2
[ 34.340393] systemd-fsck[750]: filesystem (and not swap or ufs or something else), then the superblock
[ 34.340623] systemd-fsck[750]: is corrupt, and you might try running e2fsck with an alternate superblock:
[ 34.340881] systemd-fsck[750]: e2fsck -b 8193 <device>
[ 34.343813] systemd-fsck[750]: fsck failed with error code 8.
[ 34.343824] systemd-fsck[750]: Ignoring error.
[ 34.355315] udev[764]: starting version 167
[ 34.367204] EXT3-fs (md0): error: unable to read superblock
[ 34.382275] mount[774]: mount: wrong fs type, bad option, bad superblock on /dev/md0,
[ 34.382282] mount[774]: missing codepage or helper program, or other error
[ 34.382304] mount[774]: (could this be the IDE device where you in fact use
[ 34.382311] mount[774]: ide-scsi so that sr0 or sda or so is needed?)
[ 34.382320] mount[774]: In some cases useful info is found in syslog - try
[ 34.382326] mount[774]: dmesg | tail or so
[ 34.391276] systemd[1]: var.mount mount process exited, code=exited status=32
[ 34.399827] systemd[1]: Job var-lock.mount/start failed with result 'dependency'.
[ 34.401212] systemd[1]: Job avahi-daemon.service/start failed with result 'dependency'.
[ 34.401223] systemd[1]: Job avahi-daemon.socket/start failed with result 'dependency'.
[ 34.401232] systemd[1]: Job dbus.service/start failed with result 'dependency'.
[ 34.401241] systemd[1]: Job NetworkManager.service/start failed with result 'dependency'.
[ 34.401250] systemd[1]: Job abrtd.service/start failed with result 'dependency'.
[ 34.401259] systemd[1]: Job dbus.socket/start failed with result 'dependency'.
[ 34.401267] systemd[1]: Job var-run.mount/start failed with result 'dependency'.
[ 34.401276] systemd[1]: Job fedora-autorelabel.service/start failed with result 'dependency'.
[ 34.401285] systemd[1]: Job fedora-autorelabel-mark.service/start failed with result 'dependency'.
[ 34.401294] systemd[1]: Job local-fs.target/start failed with result 'dependency'.
[ 34.401303] systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
[ 34.401312] systemd[1]: Unit var.mount entered failed state.
[ 34.705956] type=1400 audit(1310488443.085:4): avc: denied { mmap_zero } for pid=854 comm="vbetool" scontext=system_u:system_r:vbetool_t:s0-s0:c0.c1023 tcontext=system_u:system_r:vbetool_t:s0-s0:c0.c1023 tclass=memprotect
Kari, this looks like it will need more investigation. Please file a new bug and attach: - the complete output of "dmesg" after booting with "log_buf_len=1M systemd.log_level=debug systemd.log_target=kmsg" with systemd-26-8.fc15. - the same output from a boot with systemd-26-5.fc15. - The output from "lsblk" with systemd-26-5.fc15. - /etc/fstab Thanks. I rebooted to get the dmesg debug output with systemd-26-5.fc15 and experienced the same issue with mounting /dev/md0 as I had with systemd-26-7.fc15 and systemd-26-8.fc15, so I may have hardware problems or possibly cleanup problems after reverting back to systemd-26-5.fc15. I was able to restore operation after rebooting again (twice). Following the reboot, executing 'yum update --enablerepo=updates-testing systemd-26-8.fc15' resulted in a "nothing to do" message. I then executed 'yum reinstall --enablerepo=updates-testing systemd-26-8.fc15 systemd-sysv-26-8.fc15 systemd-units-26-8.fc15' and rebooted again. The reboot was successful and sendmail started on boot. At this point I don't know if it is worthwhile filing a new bug report. systemd-26-8.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report. |
Created attachment 447201 [details] dmesg output with systemd.log_level=debug systemd.log_target=kmsg Description of problem: systemd doesn't bring up sendmail on boot event though the sysvinit links are set to do so. (chkconfig --list sendmail gives 0:off 1: off 2:on 3:on 4:on 5:on 6:off). A manual start of sendmail does work (systemctl start sendmail.service). Version-Release number of selected component (if applicable): systemd.x86_64 10-1.fc14 systemd-gtk.x86_64 10-1.fc14 systemd-sysvinit.x86_64 10-1.fc14 systemd-units.x86_64 10-1.fc14 How reproducible: On every boot Steps to Reproduce: 1. enable sendmail (chkconfig sendmail on) 2. reboot Actual results: systemctl status sendmail.service: loaded, but inactive Expected results: systemctl status sendmail.service: active(running) Additional info: Machine is a vmware img. systemctl start sendmail.service does start sendmail just fine. I attached the dmesg output of a boot with systemd.log_leve=debug systemd.log_target=kmsg kernel commandline options