RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1226684 - [virt-v2v] Enable curl driver + upstream features + fixes in qemu-kvm and enable https
Summary: [virt-v2v] Enable curl driver + upstream features + fixes in qemu-kvm and ena...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm
Version: 7.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-05-31 18:40 UTC by Richard W.M. Jones
Modified: 2015-11-19 05:03 UTC (History)
9 users (show)

Fixed In Version: qemu-kvm-1.5.3-91.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-19 05:03:25 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
differences between curl.c in our 1.5.3 and curl.c upstream (29.14 KB, patch)
2015-06-01 14:27 UTC, Richard W.M. Jones 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2213 0 normal SHIPPED_LIVE qemu-kvm bug fix and enhancement update 2015-11-19 08:16:10 UTC

Description Richard W.M. Jones 2015-05-31 18:40:41 UTC 
Description of problem:

For background on this request, please see this virt-devel posting:
https://post-office.corp.redhat.com/mailman/private/virt-devel/2015-May/msg00338.html

We need to enable the curl driver in qemu-kvm, and add some
upstream features and fixes.

The patches are (from qemu-kvm-rhev):
kvm-block-curl-Improve-type-safety-of-s-timeout.patch (bug 1152901)
kvm-block.curl-adding-timeout-option.patch (bug 1132569)
kvm-curl-Allow-a-cookie-or-cookies-to-be-sent-with-http-.patch (bug 1132569)
kvm-curl-Don-t-deref-NULL-pointer-in-call-to-aio_poll.patch (bug 1132569)
kvm-curl-Add-timeout-and-cookie-options-and-misc.-fix-RH.patch (bug 1132569)

For enablement, we need to change ./configure to add:
  --enable-curl
  --block-drv-ro-whitelist=.....,https

Note virt-v2v only requires https read-only.

Version-Release number of selected component (if applicable):

qemu-kvm-1.5.3-89.el7

Additional info:

From the original description in bug 1132569:

The rewritten virt-v2v which we'll be rolling out to a limited
audience in RHEL 7.1 can convert from VMware ESX servers to
KVM and/or RHEV.  When it converts from ESX, it has to access
the source disks from the ESX datastore over https.

The previous version of virt-v2v (up to RHEL 6) had an HTTPS
client built in, so it would download the whole disk to a
temporary directory on the conversion server.  This was slow,
inefficient and required a lot of storage on the conversion
server (disk images are BIG, 100s GB / TBs not unusual).

The new version is smarter about this.  It creates a qcow2
overlay with the source as a backing file, ie. something
like this:

  qemu-img create -f qcow2 overlay -b https://esx/folder/foo

It then uses fstrim to mark parts of the backing filesystem which are
not used as discarded in the qcow2 layer.

So when it comes to do the copy/convert:

  qemu-img convert overlay destination

only the parts of the backing file which are actually in use get
copied.  Also no temporary copy needs to be made.

This is all very clever, but it does require that we enable the
curl driver in qemu-img.

Only qemu-img support is needed (not qemu), and only https is
needed (not http/ftp/all the other crap that curl can do).
Comment 2 Richard W.M. Jones 2015-06-01 14:27:39 UTC 
Created attachment 1033369 [details]
differences between curl.c in our 1.5.3 and curl.c upstream

This attachment is just FYI.

As expected quite a large set of differences, including new features.
Comment 3 Richard W.M. Jones 2015-06-01 15:37:28 UTC 
This is not as easy as it seems.  qemu 1.5.3 lacks

(1) aio timers, eg. timer_del, aio_timer_init.

These seem to be fairly fundamental to the curl block driver.
The backport of this feature seems like it could be tricky.

(2) bdrv_attach_aio_context, bdrv_detach_aio_context

Seems as if this is NOT fundamental, but would like Stefan to check.
Stefan: Is the upstream commit 63f0f45f2e89b60ff8245fec81328ddfde42a303
necessary for qemu 1.5.3?
Comment 4 Richard W.M. Jones 2015-06-01 18:06:47 UTC 
Scratch build containing the backported patches:
http://brewweb.devel.redhat.com/brew/taskinfo?taskID=9288421
Comment 5 Richard W.M. Jones 2015-06-01 20:15:23 UTC 
Improved scratch build, this time remembering to enable the
drivers in ./configure:
http://brewweb.devel.redhat.com/brew/taskinfo?taskID=9288785
Comment 6 Richard W.M. Jones 2015-06-01 20:30:08 UTC 
Using the scratch build in comment 5, I ran through a VMware vCenter
conversion which tests:

 - curl driver
    * file.cookie
    * file.readahead
    * file.sslverify
    * file.driver == https
    * file.timeout
 - copy-on-read
 - fstrim
 - json: URLs (bug 1226697)

and it was successful.
Comment 7 Richard W.M. Jones 2015-06-02 09:45:26 UTC 
http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-June/thread.html#00109
http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-June/thread.html#00132
Comment 8 Richard W.M. Jones 2015-06-02 09:56:43 UTC 
(In reply to Richard W.M. Jones from comment #7)
> http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-June/thread.
> html#00109
> http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-June/thread.
> html#00132

Ignore the above link.  Corrected patch series:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-June/thread.html#00138
http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-June/thread.html#00132
Comment 9 Stefan Hajnoczi 2015-06-08 14:45:28 UTC 
(In reply to Richard W.M. Jones from comment #3)
> (2) bdrv_attach_aio_context, bdrv_detach_aio_context
> 
> Seems as if this is NOT fundamental, but would like Stefan to check.
> Stefan: Is the upstream commit 63f0f45f2e89b60ff8245fec81328ddfde42a303
> necessary for qemu 1.5.3?

bdrv_attach_aio_context/bdrv_detach_aio_context are not needed in RHEL 7.1 qemu-kvm.

(They are needed in qemu-kvm-rhev but not qemu-kvm.)

The reason is that RHEL 7.1 qemu-kvm uses an older version of the virtio-blk data-plane code that does not use the QEMU block layer's AioContext.  Therefore attach/detach are not necessary.
Comment 10 Richard W.M. Jones 2015-06-11 11:42:22 UTC 
Spec file patch (unchanged from before):

http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-June/thread.html#00132

v3 of the curl block driver fixes:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-June/thread.html#00857
Comment 11 Miroslav Rezanina 2015-06-15 14:09:45 UTC 
Fix included in qemu-kvm-1.5.3-91.el7
Comment 13 Richard W.M. Jones 2015-06-17 16:19:32 UTC 
qemu-kvm-1.5.3-92.el7.x86_64
I've tested this now and it all seems to be working.
Comment 14 mazhang 2015-07-01 09:29:02 UTC 
(In reply to Richard W.M. Jones from comment #0)
> Description of problem:
> 
> For background on this request, please see this virt-devel posting:
> https://post-office.corp.redhat.com/mailman/private/virt-devel/2015-May/
> msg00338.html
> 
> We need to enable the curl driver in qemu-kvm, and add some
> upstream features and fixes.
> 
> The patches are (from qemu-kvm-rhev):
> kvm-block-curl-Improve-type-safety-of-s-timeout.patch (bug 1152901)
> kvm-block.curl-adding-timeout-option.patch (bug 1132569)
> kvm-curl-Allow-a-cookie-or-cookies-to-be-sent-with-http-.patch (bug 1132569)
> kvm-curl-Don-t-deref-NULL-pointer-in-call-to-aio_poll.patch (bug 1132569)
> kvm-curl-Add-timeout-and-cookie-options-and-misc.-fix-RH.patch (bug 1132569)

Hi Richard,

Since qemu-kvm-rhev also has those fixes and has been verified, may I verify this bug refer to the way of bug 1152901 and bug 1132569, is there difference between qemu-kvm-rhev and qemu-kvm?

Thanks,
Mazhang.
Comment 15 Richard W.M. Jones 2015-07-01 10:33:48 UTC 
We cannot assert that because the same patches are available upstream/
in a later version, therefore the backported patches will work.

OTOH I have tested this qemu-kvm binary myself and it works for me.
Comment 17 mazhang 2015-07-02 07:11:21 UTC 
Hi Richard,

Could you please have a look at comment#17, is it sufficient for this bug verify?

Thanks,
Mazhang.
Comment 18 Richard W.M. Jones 2015-07-02 08:07:31 UTC 
*comment 16

Yes, that is sufficient to verify it, thanks.
Comment 19 mazhang 2015-07-02 09:49:26 UTC 
Base on comment 16 and comment 18, set this bug verified.
Comment 21 errata-xmlrpc 2015-11-19 05:03:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2213.html
Note You need to log in before you can comment on or make changes to this bug.