Bug 569762 - 'qemu-img re-base' broken on block devices
Summary: 'qemu-img re-base' broken on block devices
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kvm
Version: 5.5
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Kevin Wolf
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: bonding, Bug, interface, multiple 552486 580028
TreeView+ depends on / blocked
 
Reported: 2010-03-02 10:34 UTC by Igor Lvovsky
Modified: 2013-03-01 04:51 UTC (History)
8 users (show)

Fixed In Version: kvm-83-162.el5
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 580028 (view as bug list)
Environment:
Last Closed: 2010-03-30 07:52:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2010:0271 0 normal SHIPPED_LIVE Important: kvm security, bug fix and enhancement update 2010-03-29 13:19:48 UTC

Description Igor Lvovsky 2010-03-02 10:34:15 UTC
Description of problem:
'qemu-img rebase' work fine on NFS but broken on block devices


Version-Release number of selected component (if applicable):


How reproducible:
Create chain with raw base file on block devices:

     base.raw  -> sn1.qcow2 -> sn2.qcow2

Steps to Reproduce:
1.  create raw image on block device 'base.raw'
2.  create snapshot on it 'sn1.qcow2'
3.  create snapshot on 'sn1.qcow2' 
4.  run:  qemu-img rebase -u -F raw -b base.raw  sn2.qcow2
  
Actual results:
qemu-img rebase -u -b base.raw -F raw sn2.qcow2
qemu-img: Could not change the backing file to 'base.raw': Operation not supported

Expected results:


Additional info:

Comment 5 Ayal Baron 2010-03-04 07:26:57 UTC
This is blocking bz#552486 which can lead to VM corruption.  RHEVM is dependent on this to work.
You can also see BZ#530134 which was for this exact issue for more info - the rebase feature itself is a rhel5.5 blocker and seeing as it doesn't work for block device it follows that this bug is a blocker.

Comment 8 Keqin Hong 2010-03-04 10:15:01 UTC
(In reply to comment #0)

> How reproducible:
> Create chain with raw base file on block devices:
> 
>      base.raw  -> sn1.qcow2 -> sn2.qcow2

Igor, 
Need to confirm from you one thing. As I understand both "base.raw" and "sn1.qcow2" are block devices.
Following was how I created a snapshot chain similar to the first part of yours  (simply not considering sn2.qcow2 here):
# lvs
  LV     VG     Attr   LSize  Origin Snap%  Move Log Copy%  Convert
  lv2    vgtest -wi-ao  4.00G                                      
  lv_20G vgtest -wi-a- 20.00G 
#qemu-img convert -f raw RHEL-Server-5.4-64.raw -O raw /dev/vgtest/lv_20G
#qemu-img create -b /dev/vgtest/lv_20G -f qcow2 /dev/vgtest/lv2

(/dev/vgtest/lv_20G -> /dev/vgtest/lv2)
Then, we could boot guest from /dev/vgtest/lv_20G while not from /dev/vgtest/lv2.
We got "Boot failed: not a bootable disk" on QEMU screen in the latter case.

Did I misunderstand your test environment? 
Thanks.

Comment 9 Kevin Wolf 2010-03-04 10:32:08 UTC
(In reply to comment #8)
> Then, we could boot guest from /dev/vgtest/lv_20G while not from
> /dev/vgtest/lv2.
> We got "Boot failed: not a bootable disk" on QEMU screen in the latter case.

What does your qemu-kvm command line look like? Specifically, does it contain format=qcow2 for the drive? If the image is on a block device, the format won't be autodetected.

Comment 10 Keqin Hong 2010-03-04 11:01:02 UTC
Kevin,
format=qcow2 helps, thanks.

Comment 14 Igor Lvovsky 2010-03-04 15:22:43 UTC
(In reply to comment #8)
> (In reply to comment #0)
> 
> > How reproducible:
> > Create chain with raw base file on block devices:
> > 
> >      base.raw  -> sn1.qcow2 -> sn2.qcow2
> 
> Igor, 
> Need to confirm from you one thing. As I understand both "base.raw" and
> "sn1.qcow2" are block devices.
> Following was how I created a snapshot chain similar to the first part of yours
>  (simply not considering sn2.qcow2 here):
> # lvs
>   LV     VG     Attr   LSize  Origin Snap%  Move Log Copy%  Convert
>   lv2    vgtest -wi-ao  4.00G                                      
>   lv_20G vgtest -wi-a- 20.00G 
> #qemu-img convert -f raw RHEL-Server-5.4-64.raw -O raw /dev/vgtest/lv_20G
> #qemu-img create -b /dev/vgtest/lv_20G -f qcow2 /dev/vgtest/lv2
> 
> (/dev/vgtest/lv_20G -> /dev/vgtest/lv2)
> Then, we could boot guest from /dev/vgtest/lv_20G while not from
> /dev/vgtest/lv2.
> We got "Boot failed: not a bootable disk" on QEMU screen in the latter case.
> 
> Did I misunderstand your test environment? 
> Thanks.    


I think you should use -F option for snapshot creation:
qemu-img create -F raw -b /dev/vgtest/lv_20G -f qcow2 /dev/vgtest/lv2

BTW, why you boot from base image? Once you created snapshot you should boot from it.

Comment 15 Keqin Hong 2010-03-05 01:20:03 UTC
(In reply to comment #14)
> I think you should use -F option for snapshot creation:
> qemu-img create -F raw -b /dev/vgtest/lv_20G -f qcow2 /dev/vgtest/lv2

Well, good suggestion, especially /dev/vgtest/lv_20G is a LV.

> 
> BTW, why you boot from base image? Once you created snapshot you should boot
> from it.    

I must clarify that I booted base image before creating snapshot just to test the usability of the base image.

Comment 20 Keqin Hong 2010-03-10 03:23:06 UTC
Summary:
Passed on kvm-83-162.el5 with "-f $fmt" for rebase subcmd.

Setup:

#lvs
      LV VG Attr LSize Origin Snap% Move Log Copy% Convert
      lv2 vgtest wi-a 2.00G
      lv3 vgtest wi-a 2.00G
      lv_20G vgtest wi-a 20.00G

Steps to test:
1. create snapshot chain "lv_20G (raw) <-- lv2 (qcow2) <-- lv3 (qcow2)"
#qemu-img convert -f raw RHEL-Server-5.4-64.raw -O raw /dev/vgtest/lv_20G
#qemu-img create -F raw -b /dev/vgtest/lv_20G -f qcow2 /dev/vgtest/lv2
#qemu-img create -F qcow2 -b /dev/vgtest/lv2 -f qcow2 /dev/vgtest/lv3
2. rebase
#qemu-img rebase -u -F raw -b /dev/vgtest/lv_20G -f qcow2 /dev/vgtest/lv3

Actual results:
step 2 succeeded, and after step 2 guest can be booted from /dev/vgtest/lv3

Comment 22 Keqin Hong 2010-03-10 08:10:49 UTC
Test on 8 different scenarios, the results are:
===========================================================
 NO.| snapshot chain                  | rebase -u | rebase
-----------------------------------------------------------
 1  | blk_raw <-- blk_sn1 <-- blk_sn2 | P         | P
-----------------------------------------------------------
 2  | blk_raw <-- blk_sn1 <-- sn2     | P         | P
-----------------------------------------------------------
 3  | blk_raw <-- sn1 <-- sn2         | P         | P
-----------------------------------------------------------
 4  | blk_raw <-- sn1 <-- blk_sn2     | P(-)      | F
===========================================================
 5  | blk_sn1 <-- blk_sn2 <-- blk_sn3 | P         | P
-----------------------------------------------------------
 6  | blk_sn1 <-- blk_sn2 <-- sn3     | P         | P
-----------------------------------------------------------
 7  | blk_sn1 <-- sn2 <-- sn3         | P         | P
-----------------------------------------------------------
 8  | blk_sn1 <-- blk_sn2 <-- blk_sn3 | P(-)      | F
===========================================================
blk_* means the image is on block device (e.g. a logical volume), 
sn* means it is a QCOW2 snapshot. P - pass; F - fail.

Note in case No. 4 (or 8), after creating the snapshot chain, guest can't be booted from blk_sn2, although using "rebase -u" will succeed (that's why it is marked a P(-) ).
Take for example "/dev/vgtest/lv_20G <-- sn1 <-- /dev/vgtest/lv2", if we run "qemu-img rebase -b /dev/vgtest/lv_20G -F qcow2 -f qcow2 /dev/vgtest/lv2", the result will be "qemu-img: Could not open '/dev/vgtest/lv2'". By stracing, we got
"
...
lseek(12, 0, SEEK_SET)                  = 0
read(12, "QFI\373\0\0\0\2\0\0\0\0\0\0\0X\0\0\0\3\0\0\0\20\0\0\0\5\0\0\0\0"..., 512) = 512
lseek(12, 0, SEEK_SET)                  = 0
read(12, "QFI\373\0\0\0\2\0\0\0\0\0\0\0X\0\0\0\3\0\0\0\20\0\0\0\5\0\0\0\0"..., 512) = 512
brk(0x10511000)                         = 0x10511000
open("/dev/vgtest/sn1", O_RDWR|O_DIRECT) = -1 ENOENT (No such file or directory)
brk(0x10510000)                         = 0x10510000
munmap(0x2aaaaaaab000, 1052672)         = 0
munmap(0x2aaaaabac000, 2101248)         = 0
close(12)                               = 0
write(2, "qemu: could not open disk image "..., 48qemu: could not open disk image /dev/vgtest/lv2
) = 48
exit_group(1)                           = ?
"

Kevin, does the problem lie in a wrong path resolution?
("/dev/vgtest/sn1" should be "sn1" without prefix "/dev/vgtest/")

Comment 23 Keqin Hong 2010-03-10 08:38:14 UTC
Sorry, the strace provided in Comment 22 was by booting guest from /dev/vgtest/lv2. However, using qemu-img rebase (without -u) gave similar result as in:
"
brk(0x1a69c000)                         = 0x1a69c000
lseek(5, 131072, SEEK_SET)              = 131072
read(5, "\0\0\0\0\0\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
lseek(5, 72, SEEK_SET)                  = 72
read(5, "\342y*\312\0\0\0\5", 8)        = 8
lseek(5, 80, SEEK_SET)                  = 80
read(5, "qcow2", 5)                     = 5
lseek(5, 88, SEEK_SET)                  = 88
read(5, "sn1", 3)                       = 3
open("/dev/vgtest/sn1", O_RDWR)         = -1 ENOENT (No such file or directory)
munmap(0x2aaaaaaab000, 1052672)         = 0
munmap(0x2aaaaabac000, 2101248)         = 0
close(5)                                = 0
write(2, "qemu-img: ", 10qemu-img: )              = 10
write(2, "Could not open '/dev/vgtest/lv2'", 32Could not open '/dev/vgtest/lv2') = 32
write(2, "\n", 1
)                       = 1
exit_group(1)                           = ?
"

Comment 24 Kevin Wolf 2010-03-11 08:44:27 UTC
Have you considered that the backing file path is always resolved as relative to the snapshot file itself? If you don't this, you need to use absolute paths.

If this doesn't explain your results, could you please post the exact steps you use to create the snapshot chain and the rebase command line?

Comment 25 Keqin Hong 2010-03-11 09:44:23 UTC
(In reply to comment #24)
> Have you considered that the backing file path is always resolved as relative
> to the snapshot file itself? If you don't this, you need to use absolute paths.
> 
1. Using absolute path does help.
2. By using relative path, The only way to create the backing file inside the backing folder. Take for example, if we want to create a chain like: "/root/dir1/base <-- /root/dir1/sn1 <-- /root/dir1/dir2/dir3/sn2"

we can only create sn2 by,
#pwd
/root/dir1
#cd dir2/dir3
#qemu-img create -b ../../sn1 -F qcow2 -f qcow2 sn2

and we can't do like the following, otherwise sn2 can't be started
#pwd
/root/dir1
#qemu-img create -b sn1 -F qcow2 -f qcow2 dir2/dir3/sn2


If that is the case, then the correct way to create a backing file /dev/vgtest/lv2 in a chain "/dev/vgtest/lv1 <-- /root/sn1 <-- /dev/vgtest/lv2" would be "#cd /dev/vgtest/; qemu-img create -b ../../root/sn1 -F qcow2 -f qcow2 lv2" (Yes, it works by testing).

Comment 26 Keqin Hong 2010-03-11 10:10:36 UTC
Retested case 4 and case 8 in Comment 22, both passed (using the relative path mentioned in Comment 25 to create snapshot chain).

Comment 28 Kevin Wolf 2010-03-11 10:37:57 UTC
Yes, this is how it works.

Comment 30 errata-xmlrpc 2010-03-30 07:52:47 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0271.html


Note You need to log in before you can comment on or make changes to this bug.