Bug 1302168

Summary: blockjob --pivot failed with virtlockd on after blockcopy for os images
Product: Red Hat Enterprise Linux 7 Reporter: Han Han <hhan>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: yisun
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.2CC: dyuan, jdenemar, rbalakri, xuzhang, yanyang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-3.0.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 17:09:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirtd.log none

Description Han Han 2016-01-27 01:29:21 UTC
Description of problem:
After blockcopy, do blockjob --pivot failed on guest os image due to copy file locked when virtlockd is on.

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.3.0-31.el7_2.7.x86_64
libvirt-1.2.17-13.el7_2.2.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Preparing virlockd environment
# cat /etc/libvirt/qemu-lockd.conf
auto_disk_leases = 1
require_lease_for_disks = 1
file_lockspace_dir = "/var/lib/libvirt/lockd/scsivolumes"

# cat /etc/libvirt/qemu.conf
lock_manager = "lockd"

# systemctl restart virtlockd && systemctl restart libvirtd

2. Do blockcopy&blockjob --pivot for a running guest.
# virsh blockcopy n1 vda /var/lib/libvirt/images/n1.cpy --wait --verbose
Block Copy: [100 %]
Now in mirroring phase
# virsh blockjob n1 vda --pivot
error: resource busy: Lockspace resource '873758f7fe507f7b66e4f6653fdf965b9c61ab99fa941840b22ecbc2940d7bf8' is locked

Check log and find the lockspace file belongs to n1.cpy:
2016-01-26 11:05:00.667+0000: 19216: debug : virDomainLockManagerAddImage:90 : Add disk /var/lib/libvirt/images/n1.cpy
2016-01-26 11:05:00.667+0000: 19216: debug : virLockManagerAddResource:332 : lock=0x7fa5c800ca10 type=0 name=/var/lib/libvirt/images/n1.cpy nparams=0 params=(nil) flags=0
2016-01-26 11:05:00.667+0000: 19216: debug : virLockManagerLockDaemonAddResource:569 : Using indirect lease 873758f7fe507f7b66e4f6653fdf965b9c61ab99fa941840b22ecbc2940d7bf8 for /var/lib/libvirt/images/n1.cpy
2016-01-26 11:05:00.667+0000: 19216: debug : virLockManagerAcquire:350 : lock=0x7fa5c800ca10 state='<null>' flags=0 action=0 fd=(nil)

Actual results:
As step2

Expected results:
blockjob --pivot success.

Additional info:
blockcopy&blockjob --pivot for guest image not installed os is ok.

Comment 1 Han Han 2016-01-27 01:31:12 UTC
Created attachment 1118646 [details]
libvirtd.log

Comment 3 Peter Krempa 2017-01-10 18:13:25 UTC
commit b4698538123d57808e1b904b3d06d1adaf7f3624
Author: Peter Krempa <pkrempa>
Date:   Fri Dec 16 16:06:57 2016 +0100

    qemu: blockjob: Fix locking of block copy/active block commit
    
    For the blockjobs, where libvirt is able to track the state internally
    we can fix locking of images we can remove the appropriate locks.
    
    Also when doing a pivoting operation we should not acquire the lock on
    any of those images since both are actually locked already.

Comment 5 yisun 2017-03-02 09:30:03 UTC
Verified on:
======
libvirt-3.0.0-2.el7.x86_64
qemu-kvm-rhev-2.8.0-4.el7.x86_64


PASSED


Steps:
======
1. Preparing virlockd environment
# cat /etc/libvirt/qemu-lockd.conf
auto_disk_leases = 1
require_lease_for_disks = 1
file_lockspace_dir = "/var/lib/libvirt/lockd/scsivolumes"

# cat /etc/libvirt/qemu.conf
lock_manager = "lockd"

# systemctl restart virtlockd && systemctl restart libvirtd

2. prepare a transient vm
## virsh start vm1
lDomain vm1 started

## virsh list
 Id    Name                           State
----------------------------------------------------
 11    vm1                            running


## virsh undefine vm1
Domain vm1 has been undefined

3. check the lock file
## ll /var/lib/libvirt/lockd/scsivolumes
total 0
-rw-------. 1 root root 0 Mar  2 16:45 4a871df074d9339f152adf33b7dd7b81c21d427e88009d3606aa78617a422c98

4. Do the blockcopy and check a new lock file generated for the copied file
 ## virsh blockcopy vm1 vda /var/lib/libvirt/images/vm1.cpy --wait --verbose
Block Copy: [100 %]
Now in mirroring phase

 ## virsh dumpxml vm1 | grep cpy -a10
  ...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/RHEL-7.3-latest.img'/>
      <backingStore/>
      <mirror type='file' file='/var/lib/libvirt/images/vm1.cpy' format='qcow2' job='copy' ready='yes'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/vm1.cpy'/>
      </mirror>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
...

## ll /var/lib/libvirt/lockd/scsivolumes
total 0
-rw-------. 1 root root 0 Mar  2 16:56 2bc4f0846679db12a738f2019cb3014439047f4357cccdf106e602f2855718f9
-rw-------. 1 root root 0 Mar  2 16:53 4a871df074d9339f152adf33b7dd7b81c21d427e88009d3606aa78617a422c98
5. do the blockjob pivot
## virsh blockjob vm1 vda --pivot
6. check one lock file for the original vda gone
## ll /var/lib/libvirt/lockd/scsivolumes
total 0
-rw-------. 1 root root 0 Mar  2 16:56 2bc4f0846679db12a738f2019cb3014439047f4357cccdf106e602f2855718f9
## virsh dumpxml vm1 | grep cpy -a10
  ...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm1.cpy'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
...

7. login the vm and check the os works well
## virsh console vm1
Connected to domain vm1
Escape character is ^]

Red Hat Enterprise Linux Server 7.3 Beta (Maipo)
Kernel 3.10.0-461.el7.x86_64 on an x86_64

yisun_vm1 login: root
Password:
Last login: Wed Mar  1 18:16:39 on ttyS0
[root@yisun_vm1 ~]# uname -r
3.10.0-461.el7.x86_64

Comment 6 errata-xmlrpc 2017-08-01 17:09:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 7 errata-xmlrpc 2017-08-01 23:51:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846