RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1297318 - Failed to take 2nd external snapshot for a block device
Summary: Failed to take 2nd external snapshot for a block device
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.8
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-01-11 08:02 UTC by Yang Yang
Modified: 2016-02-15 11:50 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-02-15 11:50:03 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Yang Yang 2016-01-11 08:02:56 UTC 
Description of problem:
2nd external snapshot for a block device cannot be taken, which is caused
by permission denied when creating snapshot file.

Version-Release number of selected component (if applicable):
libvirt-0.10.2-55.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.482.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. prepare a running guest with block device
<disk type='block' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native'/>
      <source dev='/dev/sdb'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

2. take 1st external snapshot
# virsh snapshot-create-as vm1 s1 --disk-only --diskspec hda,file=/var/lib/libvirt/images/vm1.s1
Domain snapshot s1 created
# virsh dumpxml vm1| grep disk -a6
<disk type='block' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native'/>
      <source dev='/var/lib/libvirt/images/vm1.s1'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

3. take 2nd external snapshot
# virsh snapshot-create-as vm1 s2 --disk-only --diskspec hda,file=/var/lib/libvirt/images/vm1.s2
error: internal error unable to execute QEMU command 'transaction': /var/lib/libvirt/images/vm1.s2: error while creating qcow2: Permission denied

Actual results:


Expected results:
external snapshot should be created without error

Additional info:
2016-01-07 02:55:46.387+0000: 31789: debug : virJSONValueToString:1133 : result={"id":"libvirt-11","error":{"class":"GenericError","desc":"/var/lib/libvirt/images/vm1.s2: error while creating qcow2: Permission denied","data":{"message":"/var/lib/libvirt/images/vm1.s2: error while creating qcow2: Permission denied"}}}
2016-01-07 02:55:46.387+0000: 31789: debug : qemuMonitorJSONCheckError:348 : unable to execute QEMU command {"execute":"transaction","arguments":{"actions":[{"type":"blockdev-snapshot-sync","data":{"device":"drive-ide0-0-0","snapshot-file":"/var/lib/libvirt/images/vm1.s2","format":"qcow2"}}]},"id":"libvirt-11"}: {"id":"libvirt-11","error":{"class":"GenericError","desc":"/var/lib/libvirt/images/vm1.s2: error while creating qcow2: Permission denied","data":{"message":"/var/lib/libvirt/images/vm1.s2: error while creating qcow2: Permission denied"}}}
2016-01-07 02:55:46.387+0000: 31789: error : qemuMonitorJSONCheckError:359 : internal error unable to execute QEMU command 'transaction': /var/lib/libvirt/images/vm1.s2: error while creating qcow2: Permission denied
Comment 2 Yang Yang 2016-01-12 01:41:54 UTC 
Pezhang from kvm team re-tested it in qemu layer, it works well

1. Boot guest (Same with above command which is used by libvirt)
/usr/libexec/qemu-kvm -name vm1 \
-M rhel6.6.0 -enable-kvm \
-m 1024 -realtime mlock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 3aa035ac-c139-80af-9164-f5a0bd920401 -nodefconfig -nodefaults \
-rtc base=utc \
-no-shutdown -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 \
-drive file=/dev/sdc,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=native \
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 \
-netdev tap,id=hostnet0 \
-device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:13:6c:1e,bus=pci.0,addr=0x3,bootindex=1 \
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 \
-vnc 127.0.0.1:1 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-msg timestamp=on \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \

2. Create 2 live snapshots
{ "execute": "transaction", "arguments": { "actions": [ { 'type': 'blockdev-snapshot-sync', 'data' : { "device": "drive-ide0-0-0", "snapshot-file": "/home/s1", "format": "qcow2" } }] } }
{"return": {}}

{ "execute": "transaction", "arguments": { "actions": [ { 'type': 'blockdev-snapshot-sync', 'data' : { "device": "drive-ide0-0-0", "snapshot-file": "/home/s2", "format": "qcow2" } }] } }
{"return": {}}

 (qemu) info block
drive-ide0-0-0: removable=0 io-status=ok file=/home/s2 backing_file=/home/s1 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0


Two Live snapshots can be created successfully
Comment 3 Han Han 2016-01-12 02:41:55 UTC 
Well, it is obvious that the permission deny error is from qemu process.
It we use gdb to attach the qemu process and catch the permission deny errno:
(gdb)b if (errno==13)
(gdb)c
When reach the breakpoint, print the backtrace:
(gdb)bt
#0  0x00007fd700d56183 in poll () from /lib64/libc.so.6
#1  0x00007fd70426cde3 in aio_poll (ctx=0x7fd705ea5ee0, blocking=true) at /usr/src/debug/qemu-kvm-0.12.1.2/aio-posix.c:230
#2  0x00007fd704260765 in bdrv_flush (bs=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:4225
#3  0x00007fd704285f96 in qcow2_cache_flush (bs=0x7fd7061f3010, c=0x7fd70618fdb0) at /usr/src/debug/qemu-kvm-0.12.1.2/block/qcow2-cache.c:162
#4  0x00007fd70427e028 in qcow2_close (bs=0x7fd7061f3010) at /usr/src/debug/qemu-kvm-0.12.1.2/block/qcow2.c:797
#5  0x00007fd704264338 in bdrv_close (bs=0x7fd7061f3010) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:1109
#6  0x00007fd704264dba in bdrv_open (bs=0x7fd7061f3010, filename=0x7ffc4e663ce0 "/var/lib/libvirt/images/vm1.s1", flags=8256, drv=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:845
#7  0x00007fd704264da7 in bdrv_open (bs=0x7fd7061e4010, filename=0x7fd7061b2fd0 "/var/lib/libvirt/images/vm1.s2", flags=8258, drv=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:843
#8  0x00007fd70427f41f in qcow2_create2 (filename=0x7fd7061b2fd0 "/var/lib/libvirt/images/vm1.s2", total_size=83886080, 
    backing_file=0x7fd7073863c0 "/var/lib/libvirt/images/vm1.s1", backing_format=0x7fd70617bc90 "qcow2", flags=74754592, cluster_size=<value optimized out>, 
    prealloc=PREALLOC_MODE_OFF) at /usr/src/debug/qemu-kvm-0.12.1.2/block/qcow2.c:1226
#9  0x00007fd70427f5a2 in qcow2_create (filename=0x7fd7061b2fd0 "/var/lib/libvirt/images/vm1.s2", options=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block/qcow2.c:1281
#10 0x00007fd7042650f4 in bdrv_img_create (filename=0x7fd7061b2fd0 "/var/lib/libvirt/images/vm1.s2", fmt=0x7fd70618fe90 "qcow2", base_filename=<value optimized out>, 
    base_fmt=0x7fd70474aa20 "p\213?\004\327\177", options=<value optimized out>, img_size=42949672960, flags=8194, errp=0x7ffc4e666238)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:4665
#11 0x00007fd70429978c in qmp_transaction (dev_list=<value optimized out>, errp=0x7ffc4e666288) at /usr/src/debug/qemu-kvm-0.12.1.2/blockdev.c:1057
#12 0x00007fd7042f856c in qmp_marshal_input_transaction (mon=<value optimized out>, qdict=<value optimized out>, ret=<value optimized out>) at rhev-qmp-marshal.c:106
#13 0x00007fd70422c590 in monitor_call_handler (mon=0x7fd706086e20, cmd=0x7fd704739cf0, params=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:4377
#14 0x00007fd70422d244 in handle_qmp_command (parser=<value optimized out>, tokens=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:5003
#15 0x00007fd704295084 in json_message_process_token (lexer=0x7fd70608b6f0, token=0x7fd706158260, type=JSON_OPERATOR, x=349, y=10)
    at /usr/src/debug/qemu-kvm-0.12.1.2/json-streamer.c:87
---Type <return> to continue, or q <return> to quit---
#16 0x00007fd704294d20 in json_lexer_feed_char (lexer=0x7fd70608b6f0, ch=125 '}', flush=false) at /usr/src/debug/qemu-kvm-0.12.1.2/json-lexer.c:303
#17 0x00007fd704294e69 in json_lexer_feed (lexer=0x7fd70608b6f0, buffer=0x7ffc4e6664b0 "}\333", <incomplete sequence \334>, size=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/json-lexer.c:355
#18 0x00007fd70422be9b in monitor_control_read (opaque=<value optimized out>, buf=<value optimized out>, size=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:5024
#19 0x00007fd7042b95aa in qemu_chr_be_write (chan=<value optimized out>, cond=<value optimized out>, opaque=0x7fd705e8ff40)
    at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-char.c:192
#20 tcp_chr_read (chan=<value optimized out>, cond=<value optimized out>, opaque=0x7fd705e8ff40) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-char.c:2286
#21 0x00007fd703676642 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#22 0x00007fd7042249e0 in glib_pollfds_poll (timeout=561) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4053
#23 main_loop_wait (timeout=561) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4079
#24 0x00007fd7042482fa in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2258
#25 0x00007fd7042293e7 in main_loop (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4273
#26 main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6731
Comment 4 Peter Krempa 2016-02-09 09:09:20 UTC 
(In reply to yangyang from comment #0)
> Steps to Reproduce:
> 1. prepare a running guest with block device
> <disk type='block' device='disk'>

So the VM has a disk type='block'.

>       <driver name='qemu' type='qcow2' cache='none' io='native'/>
>       <source dev='/dev/sdb'/>
>       <target dev='hda' bus='ide'/>
>       <alias name='ide0-0-0'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
> 
> 2. take 1st external snapshot
> # virsh snapshot-create-as vm1 s1 --disk-only --diskspec
> hda,file=/var/lib/libvirt/images/vm1.s1

This libvirt didn't backport support for changing the type of the disk during the snapshot operation, thus ...

> Domain snapshot s1 created
> # virsh dumpxml vm1| grep disk -a6
> <disk type='block' device='disk'>

After the snapshot, the disk is still type='block'

>       <driver name='qemu' type='qcow2' cache='none' io='native'/>
>       <source dev='/var/lib/libvirt/images/vm1.s1'/>

... but the source is actually a file.

>       <target dev='hda' bus='ide'/>
>       <alias name='ide0-0-0'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
> 

Could you please re-test this by using an actual block device as the snapshot target? Since libvirt doesn't support changing the target type, the snapshot target needs to stay consistent.
Comment 5 Yang Yang 2016-02-15 07:56:45 UTC 
Failed to take snapshot by using a block device as target, error as following
virsh snapshot-create yy snap-block.xml --disk-only
error: unsupported configuration: source for disk 'hda' is not a regular file; refusing to generate external snapshot name

virsh snapshot-create-as yy --disk-only --diskspec hda,file=/dev/sdc2 --no-metadata 
error: internal error unable to execute QEMU command 'transaction': /dev/sdc2: error while creating qcow2: Permission denied

steps as following
1. create 3 partition, format them to qcow2 
ll /dev/sdc*
brw-rw----. 1 root disk 8, 32 Feb 15 14:15 /dev/sdc
brw-rw----. 1 root disk 8, 33 Feb 15 15:34 /dev/sdc1
brw-rw----. 1 root disk 8, 34 Feb 15 15:34 /dev/sdc2
brw-rw----. 1 qemu qemu 8, 35 Feb 15 15:44 /dev/sdc3

2. start guest using /dev/sdc1 and install os
virsh dumpxml yy | grep disk -a6 
<disk type='block' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native'/>
      <source dev='/dev/sdc1'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

3. take snapshot by specifying snapshot xml
cat snap-block.xml 
<domainsnapshot>
<disks>
<disk name="hda" snapshot="external" type="block">
<source dev="/dev/sdc2"/>
</disk>
</disks>
</domainsnapshot>

virsh snapshot-create yy snap-block.xml --disk-only
error: unsupported configuration: source for disk 'hda' is not a regular file; refusing to generate external snapshot name

4. take snapshot by using diskspec flag
virsh snapshot-create-as yy --disk-only --diskspec hda,file=/dev/sdc2 --no-metadata 
error: internal error unable to execute QEMU command 'transaction': /dev/sdc2: error while creating qcow2: Permission denied

If I pre-create backing chain and take snapshots using reuse-external flag, it works well
steps

1. pre-create backing chain
qemu-img rebase -f qcow2 -u -b /dev/sdc1 -F qcow2 /dev/sdc2
qemu-img rebase -f qcow2 -u -b /dev/sdc2 -F qcow2 /dev/sdc3

2. create snapshot using reuse-external
virsh snapshot-create-as yy --disk-only --diskspec hda,file=/dev/sdc2 --reuse-external --no-metadata 
Domain snapshot 1455521679 created

virsh snapshot-create-as yy --disk-only --diskspec hda,file=/dev/sdc3 --reuse-external --no-metadata
Domain snapshot 1455521694 created

virsh dumpxml yy | grep disk -a6
<disk type='block' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native'/>
      <source dev='/dev/sdc3'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
Comment 6 Peter Krempa 2016-02-15 11:50:03 UTC 
(In reply to yangyang from comment #5)
> Failed to take snapshot by using a block device as target, error as following
> virsh snapshot-create yy snap-block.xml --disk-only
> error: unsupported configuration: source for disk 'hda' is not a regular
> file; refusing to generate external snapshot name
> 
> virsh snapshot-create-as yy --disk-only --diskspec hda,file=/dev/sdc2
> --no-metadata 
> error: internal error unable to execute QEMU command 'transaction':
> /dev/sdc2: error while creating qcow2: Permission denied
> 
> steps as following
> 1. create 3 partition, format them to qcow2 
> ll /dev/sdc*
> brw-rw----. 1 root disk 8, 32 Feb 15 14:15 /dev/sdc
> brw-rw----. 1 root disk 8, 33 Feb 15 15:34 /dev/sdc1
> brw-rw----. 1 root disk 8, 34 Feb 15 15:34 /dev/sdc2
> brw-rw----. 1 qemu qemu 8, 35 Feb 15 15:44 /dev/sdc3
> 
> 2. start guest using /dev/sdc1 and install os
> virsh dumpxml yy | grep disk -a6 
> <disk type='block' device='disk'>
>       <driver name='qemu' type='qcow2' cache='none' io='native'/>
>       <source dev='/dev/sdc1'/>

Support for <source dev=''> is part is not yet present, see below.

>       <target dev='hda' bus='ide'/>
>       <alias name='ide0-0-0'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
> 
> 3. take snapshot by specifying snapshot xml
> cat snap-block.xml 
> <domainsnapshot>
> <disks>
> <disk name="hda" snapshot="external" type="block">
> <source dev="/dev/sdc2"/>
> </disk>
> </disks>
> </domainsnapshot>
> 
> virsh snapshot-create yy snap-block.xml --disk-only
> error: unsupported configuration: source for disk 'hda' is not a regular
> file; refusing to generate external snapshot name
> 
> 4. take snapshot by using diskspec flag
> virsh snapshot-create-as yy --disk-only --diskspec hda,file=/dev/sdc2
> --no-metadata 
> error: internal error unable to execute QEMU command 'transaction':
> /dev/sdc2: error while creating qcow2: Permission denied

All of those are expected not to work with the code base.

> 
> If I pre-create backing chain and take snapshots using reuse-external flag,
> it works well
> steps
> 
> 1. pre-create backing chain
> qemu-img rebase -f qcow2 -u -b /dev/sdc1 -F qcow2 /dev/sdc2
> qemu-img rebase -f qcow2 -u -b /dev/sdc2 -F qcow2 /dev/sdc3
> 
> 2. create snapshot using reuse-external
> virsh snapshot-create-as yy --disk-only --diskspec hda,file=/dev/sdc2
> --reuse-external --no-metadata 
> Domain snapshot 1455521679 created
> 
> virsh snapshot-create-as yy --disk-only --diskspec hda,file=/dev/sdc3
> --reuse-external --no-metadata
> Domain snapshot 1455521694 created
> 
> virsh dumpxml yy | grep disk -a6
> <disk type='block' device='disk'>
>       <driver name='qemu' type='qcow2' cache='none' io='native'/>
>       <source dev='/dev/sdc3'/>
>       <target dev='hda' bus='ide'/>
>       <alias name='ide0-0-0'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>

This is the correct approach that I've expected to work. Thanks for testing it.

Unfortunately the upstream fix is too invasive to backport. Closing as WONTFIX.

Upstream fixed this with:

commit 7076b4b72ce65c43a88af6b11558ed37893af692
Author: Peter Krempa <pkrempa>
Date:   Tue Nov 12 14:15:51 2013 +0100

    snapshot: Add support for specifying snapshot disk backing type
    
    Add support for specifying various types when doing snapshots. This will
    later allow to do snapshots on network backed volumes. Disks of type
    'volume' are not supported by snapshots (yet).
    
    Also amend the test suite to check parsing of the various new disk
    types that can now be specified.
Note You need to log in before you can comment on or make changes to this bug.