Created attachment 920167 [details]
Collection of logfiles
Description of problem:
We use PXE-boot and dracut-network functionalities to mount a read-only iscsi-target as rootfs for a diskless workstation.
The iscsi-target is a disk partition exported via the LIO-target on a Debian machine.
The kernel and initramfs used for the PXE-boot have been created by using the mentioned rootfs, which is Fedora 20.
Since the release of the first 3.13.** kernel, we encounter the problem that the system hangs on shutdown after installing all available updates. The reported error messages seem to be related to the iscsi-initiator ("connection1:0: detected conn error (1011)"). Even after waiting several (>10) minutes, the system won't shut down.
Investigating the problem and following the advices for debugging shutdown problems, we believe that the problem is related to systemd.
Version-Release number of selected component (if applicable):
The last systemd/kernel combination that works properly for us is:
Any newer versions show the described behavior.
Steps to Reproduce:
1. Install Fedora 20 on a free disk partition
2. Provide disk partition as read-only iscsi-device
3. Run "yum remove NetworkManager" and "yum update"
4. Build initramfs for iscsi-support using dracut (hostonly=no)
5. Provide kernel and initramfs via pxelinux
6. PXE-boot client machine with iscsi target as rootfs
7. Shutdown client machine
The system hangs and won't shut down even after waiting several minutes.
The system properly shuts down after several seconds.
The problem also occurs when initiating a reboot sequence.
Currently, we use "ExecStart=/usr/bin/systemctl --force poweroff -f" in the
"/usr/lib/systemd/system/systemd-poweroff.service" file as a workaround.
sync && reboot -f
sync && poweroff -f
work as expected.
In the Fedora 20 rootfs, we use network.service instead of NetworkManager (uninstalled) to manage the network devices of the diskless workstation.
When enabling the debug-shell.service (which was active when creating the provided log files), the mentioned error message ("connection1:0: detected conn error (1011)") appears earlier, and the system eventually shuts down, usually after 3 - 4 minutes.
The logs have been collected in the serial console of a qemu virtual machine which has been PXE-booted in a virtual network (hypervisor was the Debian machine exporting the iscsi target).
The Fedora 20 installation is maintained in a virtual machine hypervised by the mentioned Debian machine. In this VM, the Fedora 20 rootfs is mounted read-write. We pass "readonlyroot" and the MAC-Adress of the boot interface to the kernel command-line of the diskless client during PXE-boot.
Created attachment 920169 [details]
Logfile of the boot process
Created attachment 920170 [details]
Command line used when creating the logs
Created attachment 920171 [details]
Created attachment 920172 [details]
Created attachment 920173 [details]
Logfile of the shutdown process
Created attachment 920174 [details]
Output of systemctl -l
Afaik "conn error (1011)" is network related problem in one form or another in the iscsi world and that not working properly can lead to a hang on shutdown anyway have you narrowed the last working systemd release the last know to exactly "systemd-208-9.fc20" since that update introduced two patches for #1026860 which then in 208-10 got removed again since the LVM rules have been updated so 208-10 should just work since it should be exactly like 208-8 ( which would point to those lvm rules being updated potentially being the culprit. )
We are currently using "systemd-208-19.fc20" and the problem persists as for various other versions between 208-19 and 208-9. Unfortunately, we can't tell if specifically 208-10 works as we don't have this version of the package available, but we will check the behavior of 208-20.
You can fetch that release ( 208-10) and other releases inbetween from koji
In the meantime we verified that everything works fine using
systemd 208-11 (208-10 was not available on koji) and systemd 208-20,
so in contrast to our initial guess, the problem seems NOT to be caused
by systemd (at least not by systemd alone).
Additionally, we found out that an update of the following packages
does NOT reproduce our problem:
so it must be related to some other package. Next, we will check the
behavior with the newest updates and when booting into the multiuser
instead of the graphical target and see what's happening.
Good news: Last thursday (2014-08-28), we installed all updates and the problem vanished.
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora 'version'
Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.
Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 20 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.
Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.
If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
Thank you for reporting this bug and we are sorry it could not be fixed.