Bug 1613110

Trying to reproduce the bug with local file like below, but the issue did not appear.

Steps:
1. Create raw file with 1T
# qemu-img create -f raw base.img 1T
Formatting 'base.img', fmt=raw size=1099511627776
2. Install rhel7.6 guest on it
/usr/libexec/qemu-kvm \
        -name 'rhel7.6' \
        -machine q35 \
        -nodefaults \
        -vga qxl \
        -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=unsafe,format=raw,file=base.img \
        -device virtio-blk-pci,id=virtio_blk_pci0,drive=drive_image1,bus=pcie.0,addr=05 \
        -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=unsafe,media=cdrom,file=RHEL7.6-Server-x86_64.iso \
        -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \
        -monitor stdio \
        -vnc :1 \
        -m 8192 \
        -smp 8 \
        -device virtio-net-pci,mac=9a:b5:b6:b1:b5:b5,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9  \
        -netdev tap,id=idxgXAlm \
        -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/timao/monitor-qmpmonitor1-20180220-094308-h9I6hRsI,server,nowait \
        -mon chardev=qmp_id_qmpmonitor1,mode=control  \
3.Create overlay
# qemu-img create -f qcow2 -o compat=1.1,backing_fmt=raw -b base.img overlay.qcow2
Formatting 'overlay.qcow2', fmt=qcow2 size=1099511627776 compat=1.1 backing_file=base.img backing_fmt=raw cluster_size=65536 lazy_refcounts=off refcount_bits=16
4. Boot the overlay, it could work normally.
5. Create one qcow2 file
# qemu-img create -f qcow2 -o compat=1.1,preallocation=off convert.qcow2 1T
Formatting 'convert.qcow2', fmt=qcow2 size=1099511627776 compat=1.1 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16
6. Convert execution
# qemu-img convert -f qcow2 -O qcow2 -n overlay.qcow2 convert.qcow2 -p
    (100.00/100%)
7. Boot VM with convert.qcow2 file, the VM works well.

(In reply to timao from comment #3)
> Trying to reproduce the bug with local file like below, but the issue did
> not appear.

When trying to reproduce the bug, please test it with the same steps reporter described. After checking the log in comment 0, you missed some information.

> 
> Steps:
> 1. Create raw file with 1T
> # qemu-img create -f raw base.img 1T
> Formatting 'base.img', fmt=raw size=1099511627776

The backing file is on NBD server(Actually the server and the client share the same host from the log). The actual size of the snapshot was 208KB, when it was created.

qemu-img 'create' '-q' '-f' 'qcow2' '-b' 'nbd:localhost:42847' '-o' 'compat=1.1,backing_fmt=raw' '/var/tmp/v2vovl72ee24.qcow2'

libguestfs: parse_json: qemu-img info JSON output:\n{\n    "backing-filename-format": "raw",\n    "virtual-size": 1000204886016,\n    "filename": "/var/tmp/v2vovl72ee24.qcow2",\n    "cluster-size": 65536,\n    "format": "qcow2",\n    "actual-size": 212992,\n    "format-specific": {\n        "type": "qcow2",\n        "data": {\n            "compat": "1.1",\n            "lazy-refcounts": false,\n            "refcount-bits": 16,\n            "corrupt": false\n        }\n    },\n    "full-backing-filename": "nbd:localhost:42847",\n    "backing-filename": "nbd:localhost:42847",\n    "dirty-flag": false\n}\n\n

> 2. Install rhel7.6 guest on it
> /usr/libexec/qemu-kvm \
>         -name 'rhel7.6' \
>         -machine q35 \
>         -nodefaults \
>         -vga qxl \
>         -drive
> id=drive_image1,if=none,snapshot=off,aio=threads,cache=unsafe,format=raw,
> file=base.img \
>         -device
> virtio-blk-pci,id=virtio_blk_pci0,drive=drive_image1,bus=pcie.0,addr=05 \
>         -drive
> id=drive_cd1,if=none,snapshot=off,aio=threads,cache=unsafe,media=cdrom,
> file=RHEL7.6-Server-x86_64.iso \
>         -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \
>         -monitor stdio \
>         -vnc :1 \
>         -m 8192 \
>         -smp 8 \
>         -device
> virtio-net-pci,mac=9a:b5:b6:b1:b5:b5,id=idMmq1jH,vectors=4,netdev=idxgXAlm,
> bus=pcie.0,addr=0x9  \
>         -netdev tap,id=idxgXAlm \
>         -chardev
> socket,id=qmp_id_qmpmonitor1,path=/var/tmp/timao/monitor-qmpmonitor1-
> 20180220-094308-h9I6hRsI,server,nowait \
>         -mon chardev=qmp_id_qmpmonitor1,mode=control  \
> 3.Create overlay
> # qemu-img create -f qcow2 -o compat=1.1,backing_fmt=raw -b base.img
> overlay.qcow2
> Formatting 'overlay.qcow2', fmt=qcow2 size=1099511627776 compat=1.1
> backing_file=base.img backing_fmt=raw cluster_size=65536 lazy_refcounts=off
> refcount_bits=16
> 4. Boot the overlay, it could work normally.

Before converting the external snapshot, the image size was increased to 278MB. 

libguestfs: parse_json: qemu-img info JSON output:\n{\n    "backing-filename-format": "raw",\n    "virtual-size": 1000204886016,\n    "filename": "/var/tmp/v2vovl72ee24.qcow2",\n    "cluster-size": 65536,\n    "format": "qcow2",\n    "actual-size": 292245504,\n    "format-specific": {\n        "type": "qcow2",\n        "data": {\n            "compat": "1.1",\n            "lazy-refcounts": false,\n            "refcount-bits": 16,\n            "corrupt": false\n        }\n    },\n    "full-backing-filename": "nbd:localhost:42847",\n    "backing-filename": "nbd:localhost:42847",\n    "dirty-flag": false\n}\n\n

> 5. Create one qcow2 file
> # qemu-img create -f qcow2 -o compat=1.1,preallocation=off convert.qcow2 1T
> Formatting 'convert.qcow2', fmt=qcow2 size=1099511627776 compat=1.1
> cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16

The destination image was in nfs backend, and it was exported by sda.

mount '10.73.224.199:/home/p2v_export' '/tmp/v2v.GsQrfs'
RHV: ESD mountpoint: /tmp/v2v.GsQrfs
RHV: ESD UUID: 8f20a49b-8ed5-4d9c-b266-71b2fc12d658
RHV: actual UID:GID of new files is 36:36
uuidgen -r
uuidgen -r
uuidgen -r
RHV: will export sda to /tmp/v2v.GsQrfs/8f20a49b-8ed5-4d9c-b266-71b2fc12d658/images/d2ddbebd-ab17-4cab-a0a6-da89098ac598/7ad39ae5-76a8-4012-9e1e-c5306e597bb1

# showmount -e 10.73.224.199
Export list for 10.73.224.199:
/home/p2v_export *
/home/p2v_data   *

> 6. Convert execution
> # qemu-img convert -f qcow2 -O qcow2 -n overlay.qcow2 convert.qcow2 -p
>     (100.00/100%)
> 7. Boot VM with convert.qcow2 file, the VM works well.

(In reply to Ping Li from comment #4)
> (In reply to timao from comment #3)
> > Trying to reproduce the bug with local file like below, but the issue did
> > not appear.
> 
> When trying to reproduce the bug, please test it with the same steps
> reporter described. After checking the log in comment 0, you missed some
> information.
> 
> > 
> > Steps:
> > 1. Create raw file with 1T
> > # qemu-img create -f raw base.img 1T
> > Formatting 'base.img', fmt=raw size=1099511627776
> 
> The backing file is on NBD server(Actually the server and the client share
> the same host from the log). The actual size of the snapshot was 208KB, when
> it was created.
> 
> qemu-img 'create' '-q' '-f' 'qcow2' '-b' 'nbd:localhost:42847' '-o'
> 'compat=1.1,backing_fmt=raw' '/var/tmp/v2vovl72ee24.qcow2'
> 
> libguestfs: parse_json: qemu-img info JSON output:\n{\n   
> "backing-filename-format": "raw",\n    "virtual-size": 1000204886016,\n   
> "filename": "/var/tmp/v2vovl72ee24.qcow2",\n    "cluster-size": 65536,\n   
> "format": "qcow2",\n    "actual-size": 212992,\n    "format-specific": {\n  
> "type": "qcow2",\n        "data": {\n            "compat": "1.1",\n         
> "lazy-refcounts": false,\n            "refcount-bits": 16,\n           
> "corrupt": false\n        }\n    },\n    "full-backing-filename":
> "nbd:localhost:42847",\n    "backing-filename": "nbd:localhost:42847",\n   
> "dirty-flag": false\n}\n\n
> 
> > 2. Install rhel7.6 guest on it
> > /usr/libexec/qemu-kvm \
> >         -name 'rhel7.6' \
> >         -machine q35 \
> >         -nodefaults \
> >         -vga qxl \
> >         -drive
> > id=drive_image1,if=none,snapshot=off,aio=threads,cache=unsafe,format=raw,
> > file=base.img \
> >         -device
> > virtio-blk-pci,id=virtio_blk_pci0,drive=drive_image1,bus=pcie.0,addr=05 \
> >         -drive
> > id=drive_cd1,if=none,snapshot=off,aio=threads,cache=unsafe,media=cdrom,
> > file=RHEL7.6-Server-x86_64.iso \
> >         -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \
> >         -monitor stdio \
> >         -vnc :1 \
> >         -m 8192 \
> >         -smp 8 \
> >         -device
> > virtio-net-pci,mac=9a:b5:b6:b1:b5:b5,id=idMmq1jH,vectors=4,netdev=idxgXAlm,
> > bus=pcie.0,addr=0x9  \
> >         -netdev tap,id=idxgXAlm \
> >         -chardev
> > socket,id=qmp_id_qmpmonitor1,path=/var/tmp/timao/monitor-qmpmonitor1-
> > 20180220-094308-h9I6hRsI,server,nowait \
> >         -mon chardev=qmp_id_qmpmonitor1,mode=control  \
> > 3.Create overlay
> > # qemu-img create -f qcow2 -o compat=1.1,backing_fmt=raw -b base.img
> > overlay.qcow2
> > Formatting 'overlay.qcow2', fmt=qcow2 size=1099511627776 compat=1.1
> > backing_file=base.img backing_fmt=raw cluster_size=65536 lazy_refcounts=off
> > refcount_bits=16
> > 4. Boot the overlay, it could work normally.
> 
> Before converting the external snapshot, the image size was increased to
> 278MB. 
> 
> libguestfs: parse_json: qemu-img info JSON output:\n{\n   
> "backing-filename-format": "raw",\n    "virtual-size": 1000204886016,\n   
> "filename": "/var/tmp/v2vovl72ee24.qcow2",\n    "cluster-size": 65536,\n   
> "format": "qcow2",\n    "actual-size": 292245504,\n    "format-specific":
> {\n        "type": "qcow2",\n        "data": {\n            "compat":
> "1.1",\n            "lazy-refcounts": false,\n            "refcount-bits":
> 16,\n            "corrupt": false\n        }\n    },\n   
> "full-backing-filename": "nbd:localhost:42847",\n    "backing-filename":
> "nbd:localhost:42847",\n    "dirty-flag": false\n}\n\n
> 
> > 5. Create one qcow2 file
> > # qemu-img create -f qcow2 -o compat=1.1,preallocation=off convert.qcow2 1T
> > Formatting 'convert.qcow2', fmt=qcow2 size=1099511627776 compat=1.1
> > cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16
> 
> The destination image was in nfs backend, and it was exported by sda.
> 
> mount '10.73.224.199:/home/p2v_export' '/tmp/v2v.GsQrfs'
> RHV: ESD mountpoint: /tmp/v2v.GsQrfs
> RHV: ESD UUID: 8f20a49b-8ed5-4d9c-b266-71b2fc12d658
> RHV: actual UID:GID of new files is 36:36
> uuidgen -r
> uuidgen -r
> uuidgen -r
> RHV: will export sda to
> /tmp/v2v.GsQrfs/8f20a49b-8ed5-4d9c-b266-71b2fc12d658/images/d2ddbebd-ab17-
> 4cab-a0a6-da89098ac598/7ad39ae5-76a8-4012-9e1e-c5306e597bb1
> 
> # showmount -e 10.73.224.199
> Export list for 10.73.224.199:
> /home/p2v_export *
> /home/p2v_data   *
> 
> > 6. Convert execution
> > # qemu-img convert -f qcow2 -O qcow2 -n overlay.qcow2 convert.qcow2 -p
> >     (100.00/100%)
> > 7. Boot VM with convert.qcow2 file, the VM works well.


Could not reproduce this issue too with nbd and nfs with the steps above.
qemu version:
qemu-img-rhev-2.12.0-10.el7.x86_64
qemu-kvm-rhev-2.12.0-10.el7.x86_64


Hi Ademar,

QE could not reproduce this issue via qemu directly, could devel help take a look of this bug which was marked as 'TestBlocker'.

Thanks.

Can you reproduce this problem on a different host? This error message is unexpected.

Another option is to grant as access to the test machine, is this possible?

(In reply to Ademar Reis from comment #6)
> Can you reproduce this problem on a different host? This error message is
> unexpected.
> 
> Another option is to grant as access to the test machine, is this possible?

Test the bug with below builds:
virt-v2v-1.38.2-11.el7.x86_64
libguestfs-1.38.2-11.el7.x86_64
libvirt-4.5.0-6.el7.x86_64
qemu-kvm-rhev-2.12.0-10.el7.x86_64
virt-p2v-1.38.2-4.el7.iso 

Steps:

Scenario1:convert mpt3sas host to libvirt by virt-p2v

1.Install rhel7.5 on mpt3sas host
2.Boot mpt3sas host into virt-p2v client
3.Input conversion server info and pass the test connection
4 Input conversion info to convert host to libvirt,then start convert 
5.The conversion could be finished without error


Scenario2:convert mpt3sas host to rhv by virt-p2v

1.Install rhel7.5 on mpt3sas host

2.Boot mpt3sas host into virt-p2v client

3.Input conversion server info and pass the test connection

4 Input conversion info to convert host to libvirt,then start convert 

5.The copying of conversion is finished but the conversion is still failed with virt-v2v: error: subprocess exited with non-zero error code 1

6.Check guestfs info

# cat /var/log/libvirt/qemu/guestfs-cdcx1xuk3p5noez0.log 
2018-08-16 07:21:27.514+0000: starting up libvirt version: 4.5.0, package: 6.el7 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2018-08-01-09:52:16, x86-017.build.eng.bos.redhat.com), qemu version: 2.12.0qemu-kvm-rhev-2.12.0-10.el7, kernel: 3.10.0-931.el7.x86_64, hostname: ibm-x3250m6-02.lab.eng.pek2.redhat.com
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none TMPDIR=/var/tmp /usr/libexec/qemu-kvm -name guest=guestfs-cdcx1xuk3p5noez0,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-56-guestfs-cdcx1xuk3p5n/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu host -m 2000 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 542613a7-6d39-4b33-8142-a626db673557 -display none -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,fd=32,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -no-acpi -boot strict=on -kernel /var/tmp/.guestfs-0/appliance.d/kernel -initrd /var/tmp/.guestfs-0/appliance.d/initrd -append 'panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=linux guestfs_identifier=v2v' -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/var/tmp/v2vovl15e12e.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=unsafe,copy-on-read=on,discard=unmap -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1,write-cache=on -drive file=/tmp/libguestfsxl0rmz/overlay1.qcow2,format=qcow2,if=none,id=drive-scsi0-0-1-0,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0,write-cache=on -chardev socket,id=charserial0,path=/tmp/libguestfsv6sIOE/console.sock -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/tmp/libguestfsv6sIOE/guestfsd.sock -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x4 -netdev user,id=usernet,net=169.254.0.0/16 -device virtio-net-pci,netdev=usernet -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
2018-08-16 07:21:27.514+0000: Domain id=56 is tainted: custom-argv
2018-08-16 07:21:27.514+0000: Domain id=56 is tainted: host-cpu
2018-08-16T07:23:02.208639Z qemu-kvm: terminating on signal 15 from pid 1412 (<unknown process>)
2018-08-16 07:23:02.408+0000: shutting down, reason=destroyed


Result:
  Seems the failure of conversion is still caused by unknown process on qemu-kvm
but the problem will happen after finish copying which is different with before (problem happened during copying)

Created attachment 1476559 [details]
virt-p2v-mpt3sas-rhv-1.38.2-11-log.zip

Hi rjones,

   Could you please help to check latest test result in comment8, I'm not sure if the bug is still caused by qemu-kvm.thanks!

The error is:

virt-v2v: changeuid: mkdir: /tmp/v2v.YJj54j/7d65058d-7b85-4064-a5f1-e2954d2990fc/master/vms/ed56492a-f0bd-4c44-b478-f04dcc04bf11: Unix.Unix_error(Unix.ENOSPC, "mkdir", "/tmp/v2v.YJj54j/7d65058d-7b85-4064-a5f1-e2954d2990fc/master/vms/ed56492a-f0bd-4c44-b478-f04dcc04bf11")

This seems to indicate that you ran out of space on the NFS
mountpoint (ie. 10.73.224.199:/home/p2v_export)

Hi mxie,

Could you please help confirm comment 11 ?

Thanks.

(In reply to CongLi from comment #12)
> Hi mxie,
> 
> Could you please help confirm comment 11 ?
> 
> Thanks.


rjones is right, the conversion could be finished without error in scenario2 of comment8 after clearing the space of NFS before converting, I also can convert MD host with latest builds successfully,so I think the bug could be closed

(In reply to mxie from comment #13)
> (In reply to CongLi from comment #12)
> > Hi mxie,
> > 
> > Could you please help confirm comment 11 ?
> > 
> > Thanks.
> 
> 
> rjones is right, the conversion could be finished without error in scenario2
> of comment8 after clearing the space of NFS before converting, I also can
> convert MD host with latest builds successfully,so I think the bug could be
> closed

Thanks mxie's confirmation, close this bug based on comment 13.