Bug 1132569

I should add, only read-only.  But the curl block driver is only
read-only in any case, although mbooth had some non-upstream patches
to enable write support.

The curl block drivers have a spotty reliability record, and are
effectively unmaintained upstream.

I figure we need initial review of block/curl.c, similar to patch
review, an understanding who's going to maintain it upstream and
downstream, and some test coverage.  Are we willing to commit the
necessary resources?  Until we tack a name to the job, we aren't ;)

Enabling the curl block drivers in the build should be as easy as
flipping configure --disable-curl in qemu-kvm.spec.  This should
provide the curl-based block drivers http, https, ftp, ftps, tftp in
the utility programs like qemu-img, but not in qemu-kvm proper,
because there drivers are further limited by configure
--block-drv-{rw,ro}-whitelist.

If we want to enable only some of the curl drivers, we need to patch
block/curl.c.

I think I _do_ need qemu to enable the https driver, since otherwise
inspection & the fstrim via libguestfs isn't going to work.

  inspection -> libguestfs appliance -> qemu -> overlay -> -b https://...

  fstrim -> libguestfs appliance -> qemu -> overlay -> -b https://...

I think I have a better handle on what we really need now.

It's:

 = block/curl.c from upstream (probably 2.1 would do)

 + https://lists.gnu.org/archive/html/qemu-devel/2014-08/msg02065.html
   block.curl: adding 'timeout' option
   [not upstream yet]

 + https://lists.gnu.org/archive/html/qemu-devel/2014-08/msg04921.html
   curl: Allow a cookie or cookies to be sent with http/https requests
   [not upstream yet]

 + https://lists.gnu.org/archive/html/qemu-devel/2014-08/msg04824.html
   curl: Don't deref NULL pointer in call to aio_poll.
   [not upstream yet, but obvious fix]

With all of that, I can get virt-v2v to run properly against a
VMware vCenter Server.

Created attachment 932642 [details]
testing patch

I'm testing the attached patch against qemu-kvm-rhev with
virt-v2v.

Fix included in qemu-kvm-rhev-2.1.2-2.el7

armbru was keen that you add:

   --enable-curl \

to the configure line, so that regressions in the build
requirements for curl are caught.  This didn't happen in the
-2 package.

Ouch, I thought this is part of your patch in 0/3. I will fix this with next build.

Missing --enable-curl included in qemu-kvm-rhev-2.1.2-3.el7

Reproduce:
Version of components:
qemu-kvm-rhev-2.1.0-3.el7.x86_64

Steps as followings:
1,
# qemu-img create -f qcow2 /home/overlay -b 'json:{"file.driver":"http", "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2", "file.readahead":"64k"}'
qemu-img: /home/overlay: Could not open 'json:{"file.driver":"http", "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2", "file.readahead":"64k"}': Unknown driver 'http': No such file or directory

Based on above show, this bz has been reproduced.

Verify:
Version of components:
qemu-kvm-rhev-2.1.2-3.el7.x86_64

Steps as followings:
1,
# qemu-img create -f qcow2 /home/overlay -b 'json:{"file.driver":"http", "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2", "file.readahead":"64k"}' 
Formatting '/home/overlay', fmt=qcow2 size=2147483648 backing_file='json:{"file.driver":"http", "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2", "file.readahead":"64k"}' encryption=off cluster_size=65536 lazy_refcounts=off 

2，
# qemu-img info /home/overlay
image: /home/overlay
file format: qcow2
virtual size: 2.0G (2147483648 bytes)
disk size: 196K
cluster_size: 65536
backing file: json:{"file.driver":"http", "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2", "file.readahead":"64k"}
Format specific information:
    compat: 1.1
    lazy refcounts: false

3，
# /usr/libexec/qemu-kvm -m 2G -monitor stdio -vnc :1 -drive file=/home/overlay,if=none,id=img,readonly=on,snapshot=on -device virtio-blk-pci,drive=img,id=sys-img
qemu-kvm: -drive file=/home/overlay,if=none,id=img,readonly=on,snapshot=on: could not open disk image /home/overlay: Could not open backing file: Driver 'http' is not whitelisted

4, check whether this guest can boot up as normal.

As step 3 show, there is still existing issues for our qemu-kvm-rhev-2.1.2-3.el7.

Hi Richard,

  As above show, QE can not verify this issue now. Could you help to deal with the above issue.

  BTW, could you give a common method to how to create https server for our routine testing. Thx.

Best Regards,
Jun Li

(In reply to Jun Li from comment #16)
> Reproduce:
> Version of components:
> qemu-kvm-rhev-2.1.0-3.el7.x86_64
> 
> Steps as followings:
> 1,
> # qemu-img create -f qcow2 /home/overlay -b 'json:{"file.driver":"http",
> "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/
> x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2", "file.readahead":"64k"}'

You need to use "file.driver" : "https"

It's correct for "file.driver" : "http" to be rejected (since we
don't want to enable http).

Also: I'm not sure if "file.readahead" : "64k" will work.  virt-v2v
specifies the size in bytes in this field.

You may also need: "file.sslverify" : "off".  But it depends on whether
you have to correct client certificates installed, and whether
fedoraproject.org is using a self-signed cert or a CA-signed cert.

(In reply to Richard W.M. Jones from comment #17)
> (In reply to Jun Li from comment #16)
> > Reproduce:
> > Version of components:
> > qemu-kvm-rhev-2.1.0-3.el7.x86_64
> > 
> > Steps as followings:
> > 1,
> > # qemu-img create -f qcow2 /home/overlay -b 'json:{"file.driver":"http",
> > "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/
> > x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2", "file.readahead":"64k"}'
> 
> You need to use "file.driver" : "https"
> 
> It's correct for "file.driver" : "http" to be rejected (since we
> don't want to enable http).
> 
> Also: I'm not sure if "file.readahead" : "64k" will work.  virt-v2v
> specifies the size in bytes in this field.
> 
> You may also need: "file.sslverify" : "off".  But it depends on whether
> you have to correct client certificates installed, and whether
> fedoraproject.org is using a self-signed cert or a CA-signed cert.

Hi Richard，

  Thanks. 

Re-verify:
Version of components:
qemu-kvm-rhev-2.1.2-3.el7.x86_64

1,
# qemu-img create -f qcow2 /home/overlay -b 'json:{"file.driver":"https", "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2", "file.sslverify":"off", "file.readahead":"64k"}' 
Formatting '/home/overlay', fmt=qcow2 size=2147483648 backing_file='json:{"file.driver":"https", "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2", "file.sslverify":"off", "file.readahead":"64k"}' encryption=off cluster_size=65536 lazy_refcounts=off


Step 2,
# qemu-img info /home/overlay
image: /home/overlay
file format: qcow2
virtual size: 2.0G (2147483648 bytes)
disk size: 196K
cluster_size: 65536
backing file: json:{"file.driver":"https", "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2", "file.sslverify":"off", "file.readahead":"64k"}
Format specific information:
    compat: 1.1
    lazy refcounts: false

Step 3,
method 1,
# /usr/libexec/qemu-kvm -m 2G -monitor stdio -vnc :1 -drive file=/home/overlay,if=none,id=img,readonly=on,snapshot=on,rerror=stop,werror=stop -device ide-drive,drive=img,id=sys-img -boot menu=on -S
OR
method 2,
# LIBGUESTFS_BACKEND=direct virt-inspector -a  /home/overlay

After step 3, use method 1 to check, guest can not boot up completely. I guess this is because the network is so bad. Using method 2, it won't return anything after 5 hours later.

So I think create a local https server is required. Could you give some suggestion? 

BTW, Also test without "file.readahead":"64k". It's the same as the above show.

I've just tried this on RHEL 7.1 (qemu-kvm-rhev-2.1.2-2.el7.x86_64).
The virt-inspector command completed in about 2 minutes.

- Make sure that http_proxy & https_proxy are not set.

- Try a server which is located more locally to you in China.

For virt-v2v we only access servers this way which are on the local
network, not half way across the world(!)

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0624.html