Once sanlock driver is enabled in libvirt's qemu.conf, libvirt will always call sanlock_register for all domains it starts even though no leases are setup for them. This is to support lease hotplug. And since vdsm enables sanlock driver, all domains should be registered with sanlockd. However, vdsm didn't enable sanlock driver until the last release (AFAIK).

So I wonder, if this could perhaps be an issue with domains started before sanlock was enabled, that is before upgrading vdsm. During the upgrade, vdsm would enable sanlock and restart libvirtd and suddenly libvirt may start querying sanlock even for domains that knew nothing about sanlock when they were started. Beware, this is just a theory that needs to be proved :-)

I reproduced the issue and can confirm that what Jiri stated in comment #10 indeed causes the problem.  Now we need to evaluate the following two approaches:

 1) either we will skip any sanlock (and related) operations for domains that were started without sanlock (or any locking mechanism)

 2) or, when starting libvirtd and getting information about running domains, we try to do the same thing we do when starting the domain (registering the PID, locking the disks, acquiring leases, whatever).

Approach (1) is indeed way easier and the only problem with it is that upgrading libvirt (or its configuration) will not get you sanlock on running domains.  Approach (2) might be able to achieve that, but it has tons of caveats.  Starting with the fact that we have no plumbing done for registering other than self PID and not ending with the fact that we don't know what to do any action while migrating fails.

Any pointers about this before I start working on one of these variants?

I would suggest option 1.  rhev/vdsm does not actually configure or use sanlock leases for vms, so libvirt is just going through the sanlock steps as a formality; they are not actually doing anything useful now (or in the near future AFAIK.)

I agree, I'll start the work ASAP.

Fixed upstream by v1.2.4-51-gd990574 and v1.2.4-52-gda879e5:

commit d99057423c4c9310cfc324d73e201eeb83c7d6b8
Author: Martin Kletzander <mkletzan>
Date:   Mon May 12 11:38:08 2014 +0200

    sanlock: code movement in virLockManagerSanlockAcquire

commit da879e592142291709f7b95b9218d32bb1869d1a
Author: Martin Kletzander <mkletzan>
Date:   Mon May 12 11:38:47 2014 +0200

    sanlock: don't fail with unregistered domains

I can reproduce this bug with libvirt-0.10.2-35.el6 and can get the expect result with the libvirt-0.10.2-36.el6, so mark this bug verified. BTW, For more detials about the reproduce steps and verify steps, you can reference the comment6 in bug 1097227
https://bugzilla.redhat.com/show_bug.cgi?id=1097227#c6

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1374.html

*** Bug 1189414 has been marked as a duplicate of this bug. ***