Bug 1147343

Summary: qemu-img create -f qcow2 overlay -b ssh://xen/ segfaults on RHEL 7
Product: Red Hat Enterprise Linux 7 Reporter: Jun Li <juli>
Component: qemu-kvm-rhevAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.1CC: juzhang, michen, mrezanin, pingl, qzhang, rjones, sherold, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.6.0-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-07 20:15:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1138359    

Description Jun Li 2014-09-29 04:43:23 UTC
Description of problem:
qemu-img will core dump when specify the half-baked url(via ssh driver).

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.1.0-5.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. create a snapshot with a non-existing storage as backing_file(via ssh driver).
# qemu-img create -f qcow2 overlay -b ssh://xen/
2.
3.

Actual results:
qemu-img will core dump.
(gdb) bt
#0  0x00007ffff6cbe0a0 in _libssh2_session_set_blocking ()
   from /usr/lib64/libssh2.so.1
#1  0x00005555555e70c3 in ssh_file_open ()
#2  0x0000555555589d13 in bdrv_open ()
#3  0x000055555558a81c in bdrv_open_image ()
#4  0x00005555555899a0 in bdrv_open ()
#5  0x000055555558b27b in bdrv_img_create ()
#6  0x000055555557affd in img_create ()
#7  0x00007ffff441daf5 in __libc_start_main ()
   from /usr/lib64/libc.so.6
#8  0x000055555557a88d in _start ()

Expected results:
No core dump.

Additional info:
Also test with glusterfs, just like followings, but no core dump.
#  qemu-img create -f qcow2 overlay -b gluster://xen/
qemu-img: overlay: Could not open 'gluster://xen/': Usage: file=gluster[+transport]://[server[:port]]/volname/image[?socket=...]: Invalid argument

Comment 2 Richard W.M. Jones 2014-09-29 07:21:47 UTC
This affects upstream qemu too:

$ ./qemu-img create -f qcow2 overlay -b 'json: { "file.driver":"ssh", "file.host":"localhost", "file.host_key_check":"no" }'
qemu-img: qobject/qdict.c:193: qdict_get_obj: Assertion `obj != ((void *)0)' failed.
Aborted

Stack trace (truncated) is:

#0  0x00007ffff0b1ec59 in raise () from /lib64/libc.so.6
#1  0x00007ffff0b20368 in abort () from /lib64/libc.so.6
#2  0x00007ffff0b17bb6 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff0b17c62 in __assert_fail () from /lib64/libc.so.6
#4  0x00005555555d50fa in qdict_get_obj (qdict=<optimized out>, 
    key=<optimized out>, type=QTYPE_QSTRING) at qobject/qdict.c:193
#5  0x00005555555d52be in qdict_get_str (qdict=<optimized out>, 
    key=<optimized out>) at qobject/qdict.c:287
#6  0x00005555555d2c0f in connect_to_ssh (s=0x555555c55bf0, 
    options=0x555555c46b80, ssh_flags=6, ssh_flags@entry=1, creat_mode=-1, 
    creat_mode@entry=0, errp=0x7fffffffd960) at block/ssh.c:528
#7  0x00005555555d337e in ssh_file_open (bs=<optimized out>, 
    options=0x555555c46b80, bdrv_flags=<optimized out>, errp=0x7fffffffd960)
    at block/ssh.c:659
#8  0x0000555555576192 in bdrv_open_common (errp=<optimized out>, 
    drv=<optimized out>, flags=<optimized out>, options=<optimized out>, 
    file=<optimized out>, bs=<optimized out>) at block.c:974
#9  bdrv_open (pbs=0x7fffffffda68, filename=0x7fffffffd960 "", 
    reference=0x555555c46040 "", options=0x555555c46b80, flags=49216, 
    drv=0x55555582b000 <bdrv_ssh>, errp=0x7fffffffda70) at block.c:1485

Comment 3 Richard W.M. Jones 2014-09-29 07:33:57 UTC
I have posted a patch on qemu-devel to fix this
problem.

Note that I could not reproduce the precise original
bug (qemu-img create -f qcow2 overlay -b ssh://xen/)
so I'm only guessing that the bug I found using a json:
URL is the same.

No link because qemu-devel mailing list archives are super-slow ...

Comment 4 Richard W.M. Jones 2014-09-29 07:41:11 UTC
Patchwork link:
https://patchwork.ozlabs.org/patch/394295/

Comment 5 Richard W.M. Jones 2014-09-29 15:23:55 UTC
v2 of the patch seems to have been accepted upstream, see:

https://lists.gnu.org/archive/html/qemu-devel/2014-09/threads.html#05525

Comment 7 Richard W.M. Jones 2015-01-19 14:05:07 UTC
Moving to RHEL 7.2, and dev-acking because we have an
upstream patch.

Comment 9 Richard W.M. Jones 2015-07-08 21:14:01 UTC
The commit:

commit 18fe46d79a6de61cb2c379fb610d834ef658d84b
Author: Richard W.M. Jones <rjones>
Date:   Mon Sep 29 09:06:22 2014 +0100

    ssh: Don't crash if either host or path is not specified.

is included in qemu-kvm >= 1.5.3-91.el7 and all recent qemu-kvm-rhev
(I think possibly even the version in RHEL 7.1), but certainly in
qemu-kvm-rhev >= 2.3.0-1.el7.

Mirek, could you add this bug to the qemu-kvm-rhev RHEL 7.2 erratum
for me please?

Comment 10 Miroslav Rezanina 2015-07-09 03:37:44 UTC
Yes, bz will be added with next errata update.

Comment 13 mazhang 2015-07-22 06:47:15 UTC
Still can hit the problem with qemu-kvm-rhev-2.3.0-10.el7, so set this bug as "assigned".

Starting program: /usr/bin/qemu-img create -f qcow2 overlay -b ssh://xen/
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff57e00a0 in _libssh2_session_set_blocking () from /lib64/libssh2.so.1
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-17.el7.x86_64 elfutils-libelf-0.160-1.el7.x86_64 elfutils-libs-0.160-1.el7.x86_64 glib2-2.40.0-4.el7.x86_64 glibc-2.17-78.el7.x86_64 glusterfs-api-3.6.0.29-2.el7.x86_64 glusterfs-libs-3.6.0.29-2.el7.x86_64 gmp-6.0.0-11.el7.x86_64 gnutls-3.3.8-12.el7.x86_64 gperftools-libs-2.4-1.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.12.2-14.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-7.el7.x86_64 libcurl-7.29.0-25.el7.x86_64 libffi-3.0.13-11.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libgcrypt-1.5.3-12.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libibverbs-1.1.8-5.el7.x86_64 libidn-1.28-3.el7.x86_64 libiscsi-1.9.0-6.el7.x86_64 libnl3-3.2.21-8.el7.x86_64 librdmacm-1.0.19.1-1.el7.x86_64 libseccomp-2.1.1-2.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libssh2-1.4.3-8.el7.x86_64 libstdc++-4.8.3-9.el7.x86_64 libtasn1-3.8-2.el7.x86_64 libunwind-1.1-3.el7.x86_64 libuuid-2.23.2-22.el7_1.x86_64 lzo-2.06-6.el7_0.2.x86_64 nettle-2.7.1-4.el7.x86_64 nspr-4.10.8-1.el7_1.x86_64 nss-3.19.1-3.el7_1.x86_64 nss-softokn-freebl-3.16.2.3-12.el7.x86_64 nss-util-3.19.1-3.el7.x86_64 openldap-2.4.39-6.el7.x86_64 openssl-libs-1.0.1e-42.el7.x86_64 p11-kit-0.20.7-3.el7.x86_64 pcre-8.32-14.el7.x86_64 snappy-1.1.0-3.el7.x86_64 systemd-libs-219-3.el7.x86_64 trousers-0.3.11.2-3.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
(gdb) bt
#0  0x00007ffff57e00a0 in _libssh2_session_set_blocking () from /lib64/libssh2.so.1
#1  0x00005555555bb7f3 in ssh_file_open (bs=<optimized out>, options=<optimized out>, bdrv_flags=<optimized out>, errp=0x7fffffffe040) at block/ssh.c:675
#2  0x0000555555577e3c in bdrv_open_common (errp=0x7fffffffe030, drv=0x55555583a8a0 <bdrv_ssh>, flags=49216, options=0x5555564d5600, file=<optimized out>, bs=0x5555564cb400)
    at block.c:1045
#3  bdrv_open (pbs=pbs@entry=0x7fffffffe148, filename=<optimized out>, filename@entry=0x5555564c4000 "ssh://xen/", reference=<optimized out>, options=0x5555564d5600, flags=49216, 
    drv=0x55555583a8a0 <bdrv_ssh>, drv@entry=0x0, errp=errp@entry=0x7fffffffe150) at block.c:1566
#4  0x00005555555789bc in bdrv_open_image (pbs=pbs@entry=0x7fffffffe148, filename=filename@entry=0x5555564c4000 "ssh://xen/", options=options@entry=0x5555564d3200, 
    bdref_key=bdref_key@entry=0x55555560acd1 "file", flags=flags@entry=49216, allow_none=allow_none@entry=true, errp=errp@entry=0x7fffffffe150) at block.c:1370
#5  0x0000555555577a70 in bdrv_open (pbs=pbs@entry=0x7fffffffe1f0, filename=filename@entry=0x5555564c4000 "ssh://xen/", reference=reference@entry=0x0, options=0x5555564d3200, 
    options@entry=0x0, flags=flags@entry=64, drv=0x0, errp=errp@entry=0x7fffffffe1e8) at block.c:1544
#6  0x00005555555791bb in bdrv_img_create (filename=filename@entry=0x7fffffffe6d0 "overlay", fmt=fmt@entry=0x7fffffffe6ca "qcow2", 
    base_filename=base_filename@entry=0x7fffffffe6db "ssh://xen/", base_fmt=base_fmt@entry=0x0, options=options@entry=0x0, img_size=<optimized out>, flags=flags@entry=64, 
    errp=errp@entry=0x7fffffffe278, quiet=quiet@entry=false) at block.c:5902
#7  0x00005555555675bd in img_create (argc=6, argv=0x7fffffffe3f0) at qemu-img.c:449
#8  0x0000555555566a1c in main (argc=7, argv=0x7fffffffe3e8) at qemu-img.c:3072

Comment 14 Richard W.M. Jones 2015-07-22 10:14:19 UTC
I'm guessing that the bug I fixed with the patch (comment 9)
was a different bug.  I am now able to reproduce the original bug,
but only on RHEL, not on Fedora.

$ qemu-img create -f qcow2 overlay -b ssh://xen/
Segmentation fault

This fails:
qemu-img-rhev-2.3.0-2.el7.x86_64

This doesn't fail:
qemu-img-2.3.0-5.fc22.x86_64

That is strange because they are almost the same version.

Comment 15 Richard W.M. Jones 2015-07-22 12:09:51 UTC
Patch posted upstream:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg311739.html

Comment 16 Richard W.M. Jones 2015-07-22 13:10:50 UTC
Patch v2 posted upstream:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg311748.html

Comment 18 Richard W.M. Jones 2015-09-09 07:42:58 UTC
This is upstream, commit 325e3904210c779a13fbbc9ee156056d045d7eee,
but the fix is not included in RHEL yet.

Setting NEEDINFO on me to post the patch.

Comment 22 Mike McCune 2016-03-28 23:28:22 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 24 Ping Li 2016-08-12 02:24:23 UTC
Reproduced the issue with qemu-kvm-rhev-2.3.0-31.el7_2.21.
# qemu-img create -f qcow2 overlay -b ssh://xen/
Segmentation fault (core dumped)

Verify it with qemu-kvm-rhev-2.6.0-19.el7. After create a snapshot with a non-existing storage as backing_file through ssh driver, core dump cannot be generated and just give error prompt about non-existing xen.
# qemu-img create -f qcow2 overlay -b ssh://xen/
qemu-img: overlay: address resolution failed for xen:22: Name or service not known

Comment 26 errata-xmlrpc 2016-11-07 20:15:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html