It took me a while to figure out what exactly is going on here. I export 2 "disks" on my system as iscsi targets through tgtd. I also have my system logging in to these, so that the are available as:
This way they have identical paths on all my systems which is handy for running virtual machines on them and migrating them.
With the new libvirt support for spice I've switched to using libvirt + virtmanager for starting my vm-s rather then a bunch of scripts. So I've added the 2 iscsi targets to my libvirt storage pool.
As said my system was configured to log in to these disks already, this is being done by the iscsi service / sysv initscript, which will log in to all
nodes in the local iscsidb which have their node.startup setting set to automatic.
After adding the 2 disks to the libvirt storage pool, my system would no longer shutdown (it would hang at the end instead). This is caused by the combination of systemd and libvirt doing some undesirable (imho) things wrt iscsi.
There are 2 different possible scenarios here:
1) The iscsi service is started first, then the libvirtd service in this case
everything works ok, the iscsi service logs into the iscsi targets, libvirt
can use them but does not touch them otherwise, and on shutdown the iscsi
service log outs of them again
2) The libvirtd service gets started first, in this case libvirt logs into
the targets itself and *changes* their c
Stopping libvirt however will not logout of the nodes (nor restore their
The changing of the node.startup setting causes the iscsi service to not
create /var/lock/subsys/iscsi because there are no nodes with a
node.startup=automatic setting, and thus their is nothing for it to do
On shutdown, libvirt does not logout of the nodes as mentioned before,
neither does the iscsi service logout because of the lack of both
/var/lock/subsys/iscsi and any targets with node.startup=automatic
So the iscsi nodes stay in a logged in state, which causes the tgtd sysv
script to exit with an error when asked to stop because their are still
user of the node at which point the system hangs with systemd waiting
indefinitely for the tgtd process to exit.
In scenario 2, there is a bug in libvirt IMHO, if it has logged into the iscsi
nodes it should also logout of them. Also it should not change the node.startup
My system was actually hitting scenario 2, because systemd seems to parallelize starting classic sysv init scripts, unless their LSB headers have Required-Start or Should-Stop headers indicating another service should be started first.
I've fixed things on my system for now by adding the following lines to the LSB header:
Which causes things to switch to scenario 1. Given the large difference in startup priorities in the pre systemd world, I assume that is how things are supposed to work. Note I also added Should-Start/Stop: tgtd in case the iscsi service is not used for doing the log in / out, and locally hosted iscsi targets are used. I needed to do the same for the iscsi init script (which I more or less maintain), otherwise the system would hang while it tried to log in before tgtd was started.
Even with these lines added to the libvirt initscript LSB header, I still believe that the scenario 2 behavior is a (separate) bug which also needs to be fixed,
Ping? After upgrading to libvirt-0.8.8-7.fc15 which overwrote my modifed libvirt init script I had a system which would not reboot / shutdown again.
At least at the:
To the LSB header, working around the issue of libvirt not logging out of nodes it logged in to, thus hanging the system on halt.
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
Hans, sorry for the lack of response. I'm pretty sure the libvirt iscsi code hasn't changed probably up through rawhide, so this may still be an issue. Reassigning to F16 for now.
Hans, can you confirm that is still an issue on F16+?
(In reply to comment #8)
> Hans, sorry for the lack of response. I'm pretty sure the libvirt iscsi code
> hasn't changed probably up through rawhide, so this may still be an issue.
> Reassigning to F16 for now.
> Hans, can you confirm that is still an issue on F16+?
I'm afraid I've moved away from using iscsi as backing for vms in the mean time, so I cannot test this (without a lot of effort).
Since there hasn't been much noise about this WRT to fedora, moving to the upstream tracker.
dallan, anyone on the libvirt team with an iscsi setup that can confirm if this issue still exists?
I'm asking around.
I just stumbled of this bugzilla entry when googl'ing for my reboot problems with Fedora 17 & 18.
I had the very same behavior (sessions of an libvirt iscsi storage pool).
What I noticed on F18 here:
During shutdown, iscsid is stopped *before* libvirtd. This prevents libvirtd from properly logging out of the scsi-target it uses. I solved this in a similar manner like the OP (systemd is used now):
Adding a startup-dependency on iscsid.service in libvirtd.service solved the problem. I'm attaching a diff of my custom /etc/systemd/system/libvirtd.service vs. the original /usr/lib/systemd/system/libvirtd.service.
BTW: The fact, that after fixing the the startup/shutdown sequence the system behaves properly, proves that the culprit is NOT libvirtd but the init-scripts (resp. service units) included in the package.
Created attachment 703811 [details]
Diff, showing the additional startup dependency.
(In reply to comment #12)
> Adding a startup-dependency on iscsid.service in libvirtd.service solved the
> problem. I'm attaching a diff of my custom
> /etc/systemd/system/libvirtd.service vs. the original
Hi Fritz, thanks for the patch; would you mind sending it to libvir-list for discussion?
(In reply to comment #14)
> (In reply to comment #12)
> > Adding a startup-dependency on iscsid.service in libvirtd.service solved the
> > problem. I'm attaching a diff of my custom
> > /etc/systemd/system/libvirtd.service vs. the original
> > /usr/lib/systemd/system/libvirtd.service.
> Hi Fritz, thanks for the patch; would you mind sending it to libvir-list for
Not at all. Just posted it there.
Appears to be accepted. See:
Will appear in 1.0.3.
Author: Fritz Elfert <firstname.lastname@example.org>
Date: Thu Feb 28 21:46:19 2013 +0100
libvirt does not logout of iscsi targets, causing system hang on shutdown
There's a quite old bug entry here:
I just stumbled over that very issue on F18. Doing a little bit
debugging of the shutdown sequence, it turns out that - at least on my
F18 installation - libvirtd is shutdown *after* iscsid, which makes it
impossible for libvirt to perform the logout of the iscsi session properly.
This patch simply adds another startup dependancy on iscsid.service
which in turn delays iscsid shutdown until after libvirtd has stopped.
Having that applied, the system shuts down properly again.
Signed-off-by: Eric Blake <email@example.com>