Bug 856247 - full RHEL 6.4 block-copy support
full RHEL 6.4 block-copy support
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.3
Unspecified Unspecified
high Severity unspecified
: rc
: ---
Assigned To: Eric Blake
Virtualization Bugs
: ZStream
Depends On: 772088 837787
Blocks: 860720 871055 888426
  Show dependency treegraph
 
Reported: 2012-09-11 10:53 EDT by Federico Simoncelli
Modified: 2013-02-21 02:23 EST (History)
13 users (show)

See Also:
Fixed In Version: libvirt-0.10.2-5.el6
Doc Type: Bug Fix
Doc Text:
Cause: RHEL 6.3 picked up an early version of block copy, prior to upstream acceptance of the feature. In the time since then, several improvements were made to the upstream feature. Consequence: Older libvirt was unable to fully manage the block copy features present in this release of qemu. Fix: The block copy code was finally approved upstream in both qemu and libvirt, and then libvirt was improved to work around the differences in the early implementation so that it can now work with all of RHEL 6.3, RHEL 6.4, and upstream qemu implementations. Result: Block copy as a means of backing up the disk images of a running domain is more powerful, and has been tested to have no regressions over the support available in RHEL 6.3.
Story Points: ---
Clone Of:
: 871055 888426 (view as bug list)
Environment:
Last Closed: 2013-02-21 02:23:25 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Federico Simoncelli 2012-09-11 10:53:19 EDT
Description of problem:
At the end of a mirroring block job (with the VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT flag) the destination image chain is not relabeled for selinux and it's not included in the appropriate cgroup.

Version-Release number of selected component (if applicable):
libvirt-0.9.10-21.el6_3.4

How reproducible:
100%

Steps to Reproduce:
1. create an image chain using block devices
2. start a VM using the just created image and issue a blockRebase (using VIR_DOMAIN_BLOCK_REBASE_COPY|VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT|VIR_DOMAIN_BLOCK_REBASE_SHALLOW)
3. complete the operation with blockJobAbort (VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT)

Actual results:
The operation fails with:

2012-09-10 12:01:54.144+0000: 2522: error : qemuMonitorJSONCheckError:338 : internal error unable to execute QEMU command '__com.redhat_drive-reopen': Could not open '/rhev/data-center/f3064767-68eb-4cac-bf87-2050539fa4a2/3d9c48da-ec64-45bb-940e-b065a4e8c150/images/9f2f5133-b44a-4375-8ecb-1e48c3dee1ee/03afd2bd-428a-457a-bee6-3978e9fa2940'

Expected results:
The operation should succeed.
Comment 2 zhe peng 2012-09-14 01:17:48 EDT
Hi Federico:

    Would you help tell me whick qemu-kvm build you used? I want to reproduce this issue on rhel6.3.z, i try qemu-kvm-rhev-0.12.1.2-2.295.el6_3.2 and qemu-kvm-0.12.1.2-2.295.el6_3.2, they all not support block jobs, thanks in advance.
Comment 3 Federico Simoncelli 2012-09-14 03:46:30 EDT
(In reply to comment #2)
> Hi Federico:
> 
>     Would you help tell me whick qemu-kvm build you used? I want to
> reproduce this issue on rhel6.3.z, i try
> qemu-kvm-rhev-0.12.1.2-2.295.el6_3.2 and qemu-kvm-0.12.1.2-2.295.el6_3.2,
> they all not support block jobs, thanks in advance.

qemu-kvm-rhev-0.12.1.2-2.312.el6.x86_64
Comment 4 zhe peng 2012-09-14 06:20:56 EDT
Hi Fedrico,

    Thanks you for your quickly response, but i still have a question about this issue, as i know, issue a blockRebase need use virsh command "blockcopy",but now,this cmd can't be used in libvirt,refer this bug: https://bugzilla.redhat.com/show_bug.cgi?id=837583, so i want to know how can reproduce this ,please give me some advice for it ,very thanks. If the feature add-in build again, i will use virsh command to try this again.
Comment 5 Federico Simoncelli 2012-09-14 07:26:03 EDT
# vgcreate vgtest1 /dev/mapper/1IET_00010001
# lvcreate -n lvbaseimg1 -L 1G vgtest1
# lvcreate -n lvbaseimg2 -L 1G vgtest1
# lvcreate -n lvleafimg1 -L 1G vgtest1
# lvcreate -n lvleafimg2 -L 1G vgtest1
# qemu-img create -f qcow2 -F raw -b /dev/vgtest1/lvbaseimg1 /dev/vgtest1/lvleafimg1
# qemu-img create -f qcow2 -F raw -b /dev/vgtest1/lvbaseimg2 /dev/vgtest1/lvleafimg2
# cat > vm1.xml <<EOF
<domain type="kvm">
  <name>vm1</name>
  <memory>131072</memory>
  <devices>
    <disk device="disk" type="block">
      <source dev="/dev/vgtest1/lvleafimg1"/>
      <target bus="ide" dev="hda"/>
      <driver name="qemu" type="qcow2"/>
    </disk>
  </devices>
  <os>
    <type arch="x86_64" machine="pc">hvm</type>
  </os>
</domain>
EOF

# for i in /dev/vgtest1/lv*; do chgrp -v qemu $(readlink -f $i); done

# virsh blockcopy --shallow --reuse-external --wait vm1 /dev/vgtest1/lvleafimg1 /dev/vgtest1/lvleafimg2
# virsh blockjob --pivot vm1 /dev/vgtest1/lvleafimg1
error: internal error unable to execute QEMU command '__com.redhat_drive-reopen': Could not open '/dev/vgtest1/lvleafimg2'
Comment 7 zhe peng 2012-09-17 06:22:31 EDT
Hi Fedrico,

     very thanks, i can reproduce this now by your steps.
build: 
  libvirt-0.9.10-21.el6_3.4.x86_64
  qemu-kvm-rhev-0.12.1.2-2.312.el6.x86_64
Comment 12 Eric Blake 2012-09-18 08:30:55 EDT
> Sadly the relabeling happens at blockcopy time instead of blockjob-pivot
> time and if the lv gets deactivated before the pivoting (there's actually a
> flow in rhev where this happens) then the switch is still failing:

Yep, I realized that overnight as well - I have to label at two different points for this to work correctly - flat label at copy start, and recursive label at pivot time.  Let me work up a v2 patch.
Comment 13 Eric Blake 2012-09-20 13:29:00 EDT
For 6.3.z:
http://post-office.corp.redhat.com/archives/rhvirt-patches/2012-September/msg00561.html

For 6.4, either this series will be picked up in time for 0.10.2 (in which case the rebase will move it into POST):
https://www.redhat.com/archives/libvir-list/2012-September/msg01122.html
or I will have to supply that series as a RHEL-only patch.
Comment 18 Dave Allan 2012-09-25 10:37:17 EDT
I talked to Eric Blake, and his changes for bug 856247 cover the libvirt side.
Comment 19 Dave Allan 2012-09-25 10:37:53 EDT
(In reply to comment #18)
> I talked to Eric Blake, and his changes for bug 856247 cover the libvirt
> side.

Urgh, wrong BZ, sorry for the noise.
Comment 22 Eric Blake 2012-09-25 15:10:21 EDT
6.3.z will be in POST with:
http://post-office.corp.redhat.com/archives/rhvirt-patches/2012-September/msg00674.html
Comment 25 Eric Blake 2012-10-15 16:30:47 EDT
The version in comment 23 needs a respin, moving back to assigned until I post the next version.
Comment 36 Eric Blake 2012-12-18 11:31:36 EST
I got another report from Federico today about a failure message of 'requires an active copy job', so there probably is a real data race that needs to be solved before the RHEL 6.4 release.  I'm looking into it further today.
Comment 37 Eric Blake 2012-12-18 11:45:13 EST
I'll go ahead and clone a new bug tracking the 'requires an active copy job' issue, to avoid holding up validation of the rest of the block-copy feature.
Comment 38 zhe peng 2013-01-04 00:36:03 EST
verify with build:
libvirt-0.10.2-14.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64

step:
# vgcreate vgtest1 /dev/mapper/1IET_00010001
# lvcreate -n lvbaseimg1 -L 1G vgtest1
# lvcreate -n lvbaseimg2 -L 1G vgtest1
# lvcreate -n lvleafimg1 -L 1G vgtest1
# lvcreate -n lvleafimg2 -L 1G vgtest1
# qemu-img create -f qcow2 -F raw -b /dev/vg_virt/lvbaseimg1 /dev/vg_virt/lvleafimg1
# qemu-img create -f qcow2 -F raw -b /dev/vg_virt/lvbaseimg2 /dev/vg_virt/lvleafimg2
# cat > vm1.xml <<EOF
<domain type="kvm">
  <name>vm1</name>
  <memory>131072</memory>
  <devices>
    <disk device="disk" type="block">
      <source dev="/dev/vg_virt/lvleafimg1"/>
      <target bus="ide" dev="hda"/>
      <driver name="qemu" type="qcow2"/>
    </disk>
  </devices>
  <os>
    <type arch="x86_64" machine="pc">hvm</type>
  </os>
</domain>
EOF

# for i in /dev/vg_virt/lv*; do chgrp -v qemu $(readlink -f $i); done

S1: 
#virsh create vm1.xml

# virsh blockcopy --shallow --reuse-external --wait vm1 /dev/vg_virt/lvleafimg1 /dev/vg_virt/lvleafimg2

Now in mirroring phase

#for i in /dev/vg_virt/lv*; do ls -Z $(readlink -f $i); done
brw-rw----. qemu qemu system_u:object_r:virt_content_t:s0 /dev/dm-0
brw-rw----. qemu qemu system_u:object_r:virt_content_t:s0 /dev/dm-1
brw-rw----. qemu qemu unconfined_u:object_r:svirt_image_t:s0:c256,c359 /dev/dm-2
brw-rw----. root root system_u:object_r:fixed_disk_device_t:s0 /dev/dm-3


make sure blockjob in phase two
#virsh blockjob --info vm1 /dev/vg_virt/lvleafimg1
Block Copy: [100 %]

# virsh blockjob --pivot vm1 /dev/vg_virt/lvleafimg1

no error display
# for i in /dev/vg_virt/lv*; do ls -Z $(readlink -f $i); done
brw-rw----. qemu qemu system_u:object_r:virt_content_t:s0 /dev/dm-0
brw-rw----. qemu qemu system_u:object_r:virt_content_t:s0 /dev/dm-1
brw-rw----. qemu qemu unconfined_u:object_r:svirt_image_t:s0:c538,c961 /dev/dm-2
brw-rw----. qemu qemu unconfined_u:object_r:svirt_image_t:s0:c538,c961 /dev/dm-3

#virsh dumpxml vm1
.....
<disk type='block' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source dev='/dev/vg_virt/lvleafimg2'/>
      <target dev='hda' bus='ide'/>
.....
worked as expect

S2:
#virsh create vm1.xml

# virsh blockcopy --shallow --reuse-external --wait vm1 /dev/vg_virt/lvleafimg1 /dev/vg_virt/lvleafimg2

Now in mirroring phase

#for i in /dev/vg_virt/lv*; do ls -Z $(readlink -f $i); done
brw-rw----. qemu qemu system_u:object_r:virt_content_t:s0 /dev/dm-0
brw-rw----. qemu qemu system_u:object_r:virt_content_t:s0 /dev/dm-1
brw-rw----. qemu qemu unconfined_u:object_r:svirt_image_t:s0:c256,c359 /dev/dm-2
brw-rw----. root root system_u:object_r:fixed_disk_device_t:s0 /dev/dm-3

# lvchange -an /dev/vg_virt/lvbaseimg2
# lvchange -ay /dev/vg_virt/lvbaseimg2

# for i in /dev/vg_virt/lv*; do ls -Z $(readlink -f $i); done
brw-rw----. qemu qemu system_u:object_r:virt_content_t:s0 /dev/dm-0
brw-rw----. root disk system_u:object_r:fixed_disk_device_t:s0 /dev/dm-1
brw-rw----. qemu qemu unconfined_u:object_r:svirt_image_t:s0:c256,c359 /dev/dm-2
brw-rw----. qemu qemu unconfined_u:object_r:svirt_image_t:s0:c256,c359 /dev/dm-3
# virsh blockjob --pivot vm1 /dev/vg_virt/lvleafimg1

no error display
# for i in /dev/vg_virt/lv*; do ls -Z $(readlink -f $i); done
brw-rw----. qemu qemu system_u:object_r:virt_content_t:s0 /dev/dm-0
brw-rw----. qemu qemu system_u:object_r:virt_content_t:s0 /dev/dm-1
brw-rw----. qemu qemu unconfined_u:object_r:svirt_image_t:s0:c256,c359 /dev/dm-2
brw-rw----. qemu qemu unconfined_u:object_r:svirt_image_t:s0:c256,c359 /dev/dm-3

#virsh dumpxml vm1
.....
<disk type='block' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source dev='/dev/vg_virt/lvleafimg2'/>
      <target dev='hda' bus='ide'/>
.....

verification passed.
Comment 39 errata-xmlrpc 2013-02-21 02:23:25 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html

Note You need to log in before you can comment on or make changes to this bug.