Bug 1413824 - virt-clone can't handle domains with rbd volumes
Summary: virt-clone can't handle domains with rbd volumes
Alias: None
Product: Virtualization Tools
Classification: Community
Component: virt-manager
Version: unspecified
Hardware: Unspecified
OS: Linux
Target Milestone: ---
Assignee: Cole Robinson
QA Contact:
Depends On:
TreeView+ depends on / blocked
Reported: 2017-01-17 04:04 UTC by Jamin W. Collins
Modified: 2020-11-03 18:21 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-11-03 18:21:45 UTC

Attachments (Terms of Use)

Description Jamin W. Collins 2017-01-17 04:04:21 UTC
Attemping to clone a domain that uses rbd volumes will result in incorrect changes being made.

Running the following command:

diff \
    -u \
    <(virsh dumpxml ${DOMAIN}) \
    <(virt-clone --print-xml --original ${DOMAIN} --name ${DOMAIN}-new |  tr \" \')

Strangely "virt-clone --printxml" and "virsh dumpxml" use different quoting characters.  Which is why there's a tr in the above command.

Resulted in diff output like the following:

--- /dev/fd/63  2017-01-17 03:50:17.162597841 +0000
+++ /dev/fd/62  2017-01-17 03:50:17.162597841 +0000

... removed correct changes to domain name and uuid ...

The following changes are completely incorrect:
@@ -21,13 +21,12 @@
-    <disk type='network' device='disk'>
+    <disk type='file' device='disk'>
       <driver name='qemu' type='raw' cache='none' io='native'/>
       <auth username='libvirt'>
         <secret type='ceph' uuid='cec0cc1a-5591-4c45-baea-6487a314b605'/>
-      <source protocol='rbd' name='vms/domain'>
-        <host name='ceph-mon-1' port='6789'/>
+      <source>
         <host name='ceph-mon-2' port='6789'/>
         <host name='ceph-mon-3' port='6789'/>

... removed correct change to MAC ...

Comment 1 Jamin W. Collins 2017-01-17 04:04:55 UTC
$ yaourt -Qo virt-clone
/usr/bin/virt-clone is owned by virt-install 1.4.0-2

Comment 2 Jamin W. Collins 2017-07-17 14:34:50 UTC
It's been a while, any update?

Comment 3 Cole Robinson 2017-07-17 14:57:33 UTC
Yes this is a known issue, virt-* still need some work to fully handle gluster/rbd/sheepdog

Comment 4 Cole Robinson 2019-06-15 18:36:54 UTC
There's been some improvements in network pool/vol handling over the past couple years. Can anyone reproduce this or a similar issue with network volumes still? If so, please attach full 'virt-clone --debug ...' or 'virt-manager --debug ...' info when reproducing.

Comment 5 Jamin W. Collins 2020-10-27 18:06:11 UTC
Sorry for the delay in responding but the changes have resulted in being unable to even run the clone portion of the command.  Though the debug output does appear to indicate that what it is doing still isn't quite correct:

$ virt-clone --debug --connect qemu:///system --print-xml --original ${DOMAIN} --name ${DOMAIN}-new
[Tue, 27 Oct 2020 18:03:28 virt-clone 1961197] DEBUG (cli:204) Launched with command line: /bin/virt-clone --debug --connect qemu:///system --print-xml --original ubuntu --name ubuntu-new
[Tue, 27 Oct 2020 18:03:28 virt-clone 1961197] DEBUG (cli:216) Requesting libvirt URI qemu:///system
[Tue, 27 Oct 2020 18:03:28 virt-clone 1961197] DEBUG (cli:219) Received libvirt URI qemu:///system
[Tue, 27 Oct 2020 18:03:28 virt-clone 1961197] DEBUG (cloner:318) Original XML:
<domain type='kvm'>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>2</vcpu>
    <type arch='x86_64' machine='pc-1.3'>hvm</type>
    <bootmenu enable='yes'/>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>qemu64</model>
  <clock offset='utc'/>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <auth username='libvirt'>
        <secret type='ceph' uuid='910ca912-727c-4e80-abbd-6f1bdc33c181'/>
      <source protocol='rbd' name='vms/ubuntu'>
        <host name='skald-1' port='6789'/>
        <host name='langhus-1' port='6789'/>
        <host name='langhus-2' port='6789'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/ubuntu-18.04.1-desktop-amd64.iso'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    <interface type='bridge'>
      <mac address='00:16:3e:54:a5:93'/>
      <source bridge='internal'/>
      <model type='virtio'/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
    <console type='pty'>
      <target type='serial' port='0'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'>
      <listen type='address'/>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

[Tue, 27 Oct 2020 18:03:28 virt-clone 1961197] DEBUG (cloner:366) Auto-generated clone name 'ubuntu-clone'
[Tue, 27 Oct 2020 18:03:30 virt-clone 1961197] DEBUG (cloner:330) Wants cloning: size=0.0 path=rbd://skald-1:6789/vms/ubuntu
[Tue, 27 Oct 2020 18:03:30 virt-clone 1961197] DEBUG (cloner:505) Cloning srcpath=rbd://skald-1:6789/vms/ubuntu dstpath=None
[Tue, 27 Oct 2020 18:03:30 virt-clone 1961197] DEBUG (cloner:528) Clone guest xml diff:
--- Original
+++ New
@@ -1,6 +1,6 @@
 <domain type="kvm">
-  <name>ubuntu</name>
-  <uuid>bb401a18-16ae-44e9-824c-99dd06533215</uuid>
+  <name>ubuntu-new</name>
+  <uuid>ee12255a-daa4-44a2-a20e-ce73864dd0b6</uuid>
   <memory unit="KiB">2097152</memory>
   <currentMemory unit="KiB">2097152</currentMemory>
   <vcpu placement="static">2</vcpu>
@@ -22,13 +22,12 @@
-    <disk type="network" device="disk">
+    <disk type="file" device="disk">
       <driver name="qemu" type="raw" cache="none" io="native"/>
       <auth username="libvirt">
         <secret type="ceph" uuid="910ca912-727c-4e80-abbd-6f1bdc33c181"/>
-      <source protocol="rbd" name="vms/ubuntu">
-        <host name="skald-1" port="6789"/>
+      <source>
         <host name="langhus-1" port="6789"/>
         <host name="langhus-2" port="6789"/>
@@ -54,7 +53,7 @@
       <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
     <interface type="bridge">
-      <mac address="00:16:3e:54:a5:93"/>
+      <mac address="52:54:00:ec:e3:67"/>
       <source bridge="internal"/>
       <model type="virtio"/>
       <boot order="2"/>

[Tue, 27 Oct 2020 18:03:30 virt-clone 1961197] DEBUG (cli:255)   File "/bin/virt-clone", line 8, in <module>
  File "/usr/share/virt-manager/virtinst/virtclone.py", line 214, in runcli
  File "/usr/share/virt-manager/virtinst/cli.py", line 255, in fail

[Tue, 27 Oct 2020 18:03:30 virt-clone 1961197] ERROR (cli:256) expected str, bytes or os.PathLike object, not NoneType
[Tue, 27 Oct 2020 18:03:30 virt-clone 1961197] DEBUG (cli:258) 
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtinst/virtclone.py", line 208, in runcli
  File "/usr/share/virt-manager/virtinst/virtclone.py", line 191, in main
  File "/usr/share/virt-manager/virtinst/virtclone.py", line 65, in _validate_disks
  File "/usr/share/virt-manager/virtinst/cli.py", line 376, in validate_disk
  File "/usr/share/virt-manager/virtinst/cli.py", line 369, in check_size_conflict
    isfatal, errmsg = dev.is_size_conflict()
  File "/usr/share/virt-manager/virtinst/devices/disk.py", line 794, in is_size_conflict
    return self._storage_backend.is_size_conflict()
  File "/usr/share/virt-manager/virtinst/diskbackend.py", line 520, in is_size_conflict
    vfs = os.statvfs(os.path.dirname(os.path.abspath(self._path)))
  File "/usr/lib/python3.8/posixpath.py", line 374, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

Comment 6 Cole Robinson 2020-11-03 18:21:45 UTC
Thanks for the follow up.

In your setup, are you using libvirt storage pools for the rbd volume? If not, there's no chance this will ever work with virt-clone. libvirt storage pools are how we do all storage provisioning, and if they aren't in the mix then this never has a chance of working. I'm stating this up front to set expectations for the future.

Now even if you are using libvirt storage pools I'm guessing we are still making mistakes here. Please provide your rbd storage pool XML and and the storage XML matching the storage volume that the VM XML mentioned above is using.

Those issues aside, I pushed two changes upstream to make this less ambiguous:

1) The command you are using with --print-xml is incomplete. virt-clone should be exiting if --file or --auto-clone isn't specified, even for --print-xml. --auto-clone is the explicit 'please fill in defaults' switch. That explains some of the weird XML change behavior you are seeing. I pushed a change that explicitly errors in this case.

2) The other thing is that disk type='network' volumes that aren't backed by storage volumes should be explicitly rejected as clone targets. I added this upstream.

It sounds like in the mean time you want to use 'virt-clone --print-xml' to make the non-storage XML changes, and then you fix these up by hand by correcting the storage path. You can still get similar behavior by adding the --preserve-data option, which will leave your disk XML unchanged from the source XML.

If you have follow up issues or want to discuss more, please open an issue in the github tracker. We are no longer using bugzilla for upstream issues: https://github.com/virt-manager/virt-manager/issues

Note You need to log in before you can comment on or make changes to this bug.