Bug 1745209

Summary: qemu-img gets stuck when stream-converting from http
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Ademar Reis <areis>
Component: qemu-kvmAssignee: Hanna Czenczek <hreitz>
Status: CLOSED ERRATA QA Contact: Tingting Mao <timao>
Severity: medium Docs Contact:
Priority: high    
Version: 8.1CC: awels, cnv-qe-bugs, coli, grajaiya, hreitz, jinzhao, juzhang, knoel, ncredi, ngavrilo, rbalakri, sgordon, timao, virt-maint, ycui
Target Milestone: rcKeywords: Regression
Target Release: 8.1Flags: knoel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-4.1.0-16.module+el8.1.1+4917+752cfd65 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1744602 Environment:
Last Closed: 2020-02-04 18:28:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1744602, 1758964    

Description Ademar Reis 2019-08-23 22:09:41 UTC
+++ This bug was initially created as a clone of Bug #1744602 +++

...

--- Additional comment from Max Reitz on 2019-08-22 15:05:19 EDT ---

It’s just that it hangs later (somewhere between 40 and 80 %).  By bisecting, I found that this series is responsible: https://lists.nongnu.org/archive/html/qemu-block/2018-11/msg00011.html

That series has refactored the qcow2 decompression code to use threading, so it’s clear that the problem in the curl code has not been fixed.

(I haven’t found the cause of the hang yet.)

Max

...

--- Additional comment from Tingting Mao on 2019-08-23 04:18:48 EDT ---

Hi,


In 'qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93', convert will hit I/O error.

# qemu-img convert -p -O raw https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img tgt.img
qemu-img: curl: Operation timed out after 5000 milliseconds with 152000 out of 268278 bytes received
qemu-img: curl: Operation timed out after 5000 milliseconds with 152000 out of 327732 bytes received
qemu-img: curl: Operation timed out after 4913 milliseconds with 312000 out of 327575 bytes received
qemu-img: curl: Operation timed out after 5001 milliseconds with 393216 out of 655360 bytes received
qemu-img: curl: Operation timed out after 5000 milliseconds with 147456 out of 393216 bytes received
qemu-img: curl: Operation timed out after 5000 milliseconds with 163840 out of 327680 bytes received
qemu-img: curl: Operation timed out after 5000 milliseconds with 112000 out of 325812 bytes received
qemu-img: curl: Operation timed out after 5001 milliseconds with 56000 out of 327543 bytes received
qemu-img: error while reading sector 26624: Input/output error
qemu-img: curl: Operation timed out after 5001 milliseconds with 229376 out of 327680 bytes received
qemu-img: curl: Operation timed out after 5000 milliseconds with 393216 out of 524288 bytes received
qemu-img: error while reading sector 27776: Input/output error
qemu-img: error while reading sector 27904: Input/output error


In "qemu-kvm-2.12.0-85.module+el8.1.0+4010+d6842f29", convert works normally.

# qemu-img convert -p -O raw https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img tgt.img 
    (100.00/100%)


In 'qemu-kvm-3.1.0-29.module+el8.0.1+3728+47706820', convert will hit core dumped.

# qemu-img convert -p -O raw https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img tgt.img 
Segmentation fault (core dumped)

(gdb) bt
#0  0x00007f1d1a184858 in curl_multi_do_locked (s=<optimized out>, s=<optimized out>) at block/curl.c:414
#1  0x00007f1d1a1848e6 in curl_multi_do (arg=0x563d12409620) at block/curl.c:426
#2  0x0000563d12123862 in aio_dispatch_handlers (ctx=ctx@entry=0x563d123edf40) at util/aio-posix.c:421
#3  0x0000563d121244cf in aio_poll (ctx=ctx@entry=0x563d123edf40, blocking=blocking@entry=true)
    at util/aio-posix.c:707
#4  0x0000563d120b0c45 in blk_prw (blk=blk@entry=0x563d124bcc20, offset=offset@entry=0, 
    buf=buf@entry=0x7ffd65cf1190 " nC\022=V", bytes=bytes@entry=512, 
    co_entry=co_entry@entry=0x563d120b2720 <blk_read_entry>, flags=flags@entry=0)
    at block/block-backend.c:1262
#5  0x0000563d120b1eca in blk_pread (blk=blk@entry=0x563d124bcc20, offset=offset@entry=0, 
    buf=buf@entry=0x7ffd65cf1190, count=count@entry=512) at block/block-backend.c:1424
#6  0x0000563d1207b7a9 in find_image_format (file=file@entry=0x563d124bcc20, 
    filename=filename@entry=0x7ffd65cf34d1 "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", pdrv=pdrv@entry=0x7ffd65cf1410, errp=errp@entry=0x7ffd65cf1418) at block.c:725
#7  0x0000563d12082f9d in bdrv_open_inherit (
    filename=filename@entry=0x7ffd65cf34d1 "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", reference=reference@entry=0x0, options=0x563d123f6e50, options@entry=0x563d123f1800, 
    flags=<optimized out>, flags@entry=0, parent=parent@entry=0x0, child_role=child_role@entry=0x0, 
    errp=0x7ffd65cf1500) at block.c:2787
#8  0x0000563d12083d55 in bdrv_open (
    filename=filename@entry=0x7ffd65cf34d1 "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", reference=reference@entry=0x0, options=options@entry=0x563d123f1800, flags=flags@entry=0, 
    errp=errp@entry=0x7ffd65cf1500) at block.c:2911
#9  0x0000563d120b3935 in blk_new_open (
    filename=0x7ffd65cf34d1 "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", 
    reference=0x0, options=0x563d123f1800, flags=0, errp=0x7ffd65cf1500) at block/block-backend.c:379
#10 0x0000563d1207465b in img_open_file (
    filename=0x7ffd65cf34d1 "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", 
    options=0x563d123f1800, fmt=<optimized out>, flags=0, writethrough=<optimized out>, 
    force_share=<optimized out>, quiet=<optimized out>) at qemu-img.c:325
#11 0x0000563d12074832 in img_open (image_opts=<optimized out>, 
--Type <RET> for more, q to quit, c to continue without paging--
    filename=0x7ffd65cf34d1 "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", 
    fmt=<optimized out>, flags=0, writethrough=<optimized out>, quiet=<optimized out>, force_share=false)
    at qemu-img.c:370
#12 0x0000563d12079020 in img_convert (argc=<optimized out>, argv=0x7ffd65cf1990) at qemu-img.c:2218
#13 0x0000563d12071ebd in main (argc=6, argv=<optimized out>) at qemu-img.c:4976

Comment 1 Ademar Reis 2019-08-28 14:46:01 UTC
Patches posted: [Qemu-devel] [PATCH 0/6] block/curl: Fix hang and potential crash

We'll try a RHEL-AV-8.1.0 backport.

Comment 6 Tingting Mao 2019-11-28 09:17:54 UTC
Tried to verify this bug as below. Set this bug as verified accordingly.


Tested with:
qemu-kvm-4.1.0-16.module+el8.1.1+4917+752cfd65
kernel-4.18.0-147.el8.x86_64


Steps:
# qemu-img convert -O raw 'json:{"file.driver":"https", "file.url":"https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", "file.sslverify":"off", "file.readahead":"64k", "file.timeout": 15}' tgt.img -p
    (100.00/100%)


Result:
Convert successfully with timeout=15.



While in 'qemu-kvm-4.1.0-14.module+el8.1.0+4548+ed1300f4', failed to convert even with timeout=100.

# qemu-img convert -O raw 'json:{"file.driver":"https", "file.url":"https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", "file.sslverify":"off", "file.readahead":"64k", "file.timeout": 100}' tgt.img -p
Segmentation fault (core dumped)

Comment 8 errata-xmlrpc 2020-02-04 18:28:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0404