Created attachment 920167 [details] Collection of logfiles Description of problem: We use PXE-boot and dracut-network functionalities to mount a read-only iscsi-target as rootfs for a diskless workstation. The iscsi-target is a disk partition exported via the LIO-target on a Debian machine. The kernel and initramfs used for the PXE-boot have been created by using the mentioned rootfs, which is Fedora 20. Since the release of the first 3.13.** kernel, we encounter the problem that the system hangs on shutdown after installing all available updates. The reported error messages seem to be related to the iscsi-initiator ("connection1:0: detected conn error (1011)"). Even after waiting several (>10) minutes, the system won't shut down. Investigating the problem and following the advices for debugging shutdown problems, we believe that the problem is related to systemd. Version-Release number of selected component (if applicable): The last systemd/kernel combination that works properly for us is: systemd-208-9.fc20 kernel-3.12.8-300.fc20.x86_64 Any newer versions show the described behavior. How reproducible: always Steps to Reproduce: 1. Install Fedora 20 on a free disk partition 2. Provide disk partition as read-only iscsi-device 3. Run "yum remove NetworkManager" and "yum update" 4. Build initramfs for iscsi-support using dracut (hostonly=no) 5. Provide kernel and initramfs via pxelinux 6. PXE-boot client machine with iscsi target as rootfs 7. Shutdown client machine Actual results: The system hangs and won't shut down even after waiting several minutes. Expected results: The system properly shuts down after several seconds. Additional info: The problem also occurs when initiating a reboot sequence. Currently, we use "ExecStart=/usr/bin/systemctl --force poweroff -f" in the "/usr/lib/systemd/system/systemd-poweroff.service" file as a workaround. The commands sync && reboot -f sync && poweroff -f work as expected. In the Fedora 20 rootfs, we use network.service instead of NetworkManager (uninstalled) to manage the network devices of the diskless workstation. When enabling the debug-shell.service (which was active when creating the provided log files), the mentioned error message ("connection1:0: detected conn error (1011)") appears earlier, and the system eventually shuts down, usually after 3 - 4 minutes. The logs have been collected in the serial console of a qemu virtual machine which has been PXE-booted in a virtual network (hypervisor was the Debian machine exporting the iscsi target). The Fedora 20 installation is maintained in a virtual machine hypervised by the mentioned Debian machine. In this VM, the Fedora 20 rootfs is mounted read-write. We pass "readonlyroot" and the MAC-Adress of the boot interface to the kernel command-line of the diskless client during PXE-boot.
Created attachment 920169 [details] Logfile of the boot process
Created attachment 920170 [details] Command line used when creating the logs
Created attachment 920171 [details] dmesg logs
Created attachment 920172 [details] messages logfile
Created attachment 920173 [details] Logfile of the shutdown process
Created attachment 920174 [details] Output of systemctl -l
Afaik "conn error (1011)" is network related problem in one form or another in the iscsi world and that not working properly can lead to a hang on shutdown anyway have you narrowed the last working systemd release the last know to exactly "systemd-208-9.fc20" since that update introduced two patches for #1026860 which then in 208-10 got removed again since the LVM rules have been updated so 208-10 should just work since it should be exactly like 208-8 ( which would point to those lvm rules being updated potentially being the culprit. )
We are currently using "systemd-208-19.fc20" and the problem persists as for various other versions between 208-19 and 208-9. Unfortunately, we can't tell if specifically 208-10 works as we don't have this version of the package available, but we will check the behavior of 208-20.
You can fetch that release ( 208-10) and other releases inbetween from koji http://koji.fedoraproject.org/koji/packageinfo?packageID=10477
In the meantime we verified that everything works fine using systemd 208-11 (208-10 was not available on koji) and systemd 208-20, so in contrast to our initial guess, the problem seems NOT to be caused by systemd (at least not by systemd alone). Additionally, we found out that an update of the following packages does NOT reproduce our problem: audit,clutter,dbus,device-mapper,dhclient,dracut,gtk3,ibus,initsctipts, kernel,libmount,libuuid,mutter,mutter-wayland,selinux-policy,xorg-x11, iscsi-initiator-utils-iscsiuio so it must be related to some other package. Next, we will check the behavior with the newest updates and when booting into the multiuser instead of the graphical target and see what's happening.
Good news: Last thursday (2014-08-28), we installed all updates and the problem vanished.
This message is a reminder that Fedora 20 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 20. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '20'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 20 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.