Bug 984861

Summary: qemu core dump with chardev: opening backend "socket"/"file" failed
Product: Red Hat Enterprise Linux 6 Reporter: Sibiao Luo <sluo>
Component: qemu-kvmAssignee: Amit Shah <amit.shah>
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.5CC: acathrow, amit.shah, bsarathy, chayang, flang, juzhang, mazhang, mdeng, michen, mkenneth, qzhang, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-01 23:38:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sibiao Luo 2013-07-16 08:33:57 UTC
Description of problem:
Boot guest with virtio serial wich file chardev backend, then opening backend "file" failed and QEMU will Segmentation fault (core dumped).

Version-Release number of selected component (if applicable):
host info:
kernel-2.6.32-398.el6.x86_64
qemu-kvm-0.12.1.2-2.378.el6.x86_64
guest info:
kernel-2.6.32-398.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Boot guest with virtio serial wich file chardev backend.
e.g:...-device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev file,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev file,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2
2.
3.

Actual results:
after step 1, opening backend "file" failed and QEMU will Segmentation fault (core dumped).

chardev: opening backend "file" failed
Segmentation fault (core dumped)

(gdb) bt
#0  0x00007faf3cc4ba9e in quit_timers () at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1858
#1  0x00007faf3a58bda2 in exit () from /lib64/libc.so.6
#2  0x00007faf3cc50851 in select_vgahw (argc=60, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4563
#3  main (argc=60, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:5661
(gdb) bt full
#0  0x00007faf3cc4ba9e in quit_timers () at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1858
No locals.
#1  0x00007faf3a58bda2 in exit () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007faf3cc50851 in select_vgahw (argc=60, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4563
        opts = 0x7fffee0162af ""
#3  main (argc=60, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:5661
        popt = <value optimized out>
        gdbstub_dev = <value optimized out>
        i = <value optimized out>
        snapshot = <value optimized out>
        linux_boot = <value optimized out>
        initrd_filename = <value optimized out>
        kernel_filename = <value optimized out>
        kernel_cmdline = <value optimized out>
        boot_devices = "cad", '\000' <repeats 29 times>
        ds = <value optimized out>
        dcl = <value optimized out>
        cyls = <value optimized out>
        heads = <value optimized out>
        secs = <value optimized out>
        translation = <value optimized out>
        hda_opts = <value optimized out>
        opts = <value optimized out>
        olist = <value optimized out>
        optind = 60
        optarg = 0x7fffee01688a "stdio"
        loadvm = <value optimized out>
        machine = <value optimized out>
        cpu_model = <value optimized out>
        fds = {1019027456, 32687}
        tb_size = <value optimized out>
        pid_file = <value optimized out>
        incoming = <value optimized out>
        fd = <value optimized out>
        pwd = <value optimized out>
        chroot_dir = <value optimized out>
        run_as = <value optimized out>
        env = <value optimized out>
        show_vnc_port = <value optimized out>
        defconfig = <value optimized out>
        defconfig_verbose = <value optimized out>
(gdb)

Expected results:
it should no core dump, and can write some data to serial port inside guest which work well.

Additional info:

Comment 1 Sibiao Luo 2013-07-16 08:34:58 UTC
# /usr/libexec/qemu-kvm -S -M rhel6.5.0 -cpu SandyBridge -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1 -no-kvm-pit-reinjection -name sluo -uuid 43425b70-86e5-4664-bf2c-3b76699b8bec -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev file,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev file,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=/home/RHEL6.5-20130712.n.0-6.5-Server-x64.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK1" -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-system-disk,id=system-disk,bootindex=1 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x5 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=2C:41:38:B6:40:21,bus=pci.0,addr=0x6,bootindex=2 -drive file=/home/my-data-disk.qcow2,if=none,id=drive-data-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK2" -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-data-disk,id=data-disk,bootindex=3 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice port=5931,disable-ticketing -monitor stdio

Comment 2 Sibiao Luo 2013-07-22 04:45:16 UTC
Also met the same core dump bt log when boot up the guest with an chardev(unix socket, opened) and qmp monitor if there is no listen by nc -lU /path/to/unix-socket first. Could you help me check if is the same issue, I will separate to a new bug if not.

Version-Release number of selected component (if applicable):
host info:
kernel-2.6.32-398.el6.x86_64
qemu-kvm-0.12.1.2-2.378.el6.x86_64
guest info:
kernel-2.6.32-398.el6.x86_64

How reproducible:
100%

steps:
1.*without* listen by nc -lU /path/to/unix-socket first.
2..boot a guest by two ways:           
 a.-chardev socket,id=mon,path=/opt/mon.monitor -mon chardev=mon,mode=control            
 b. -qmp unix:/opt/mon.monitor

Exact Results:
after step 2, qemu will quit and core dumped,  like:
connect(unix:/opt/mon.monitor): No such file or directory
chardev: opening backend "socket" failed
Segmentation fault (core dumped)
(gdb) bt
#0  0x00007fb1c890ca9e in quit_timers () at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1858
#1  0x00007fb1c624cda2 in exit () from /lib64/libc.so.6
#2  0x00007fb1c8911851 in select_vgahw (argc=52, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4563
#3  main (argc=52, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:5661
(gdb) bt full
#0  0x00007fb1c890ca9e in quit_timers () at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1858
No locals.
#1  0x00007fb1c624cda2 in exit () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007fb1c8911851 in select_vgahw (argc=52, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4563
        opts = 0x7fffed7ff3e6 ""
#3  main (argc=52, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:5661
        popt = <value optimized out>
        gdbstub_dev = <value optimized out>
        i = <value optimized out>
        snapshot = <value optimized out>
        linux_boot = <value optimized out>
        initrd_filename = <value optimized out>
        kernel_filename = <value optimized out>
        kernel_cmdline = <value optimized out>
        boot_devices = "cad", '\000' <repeats 29 times>
        ds = <value optimized out>
        dcl = <value optimized out>
        cyls = <value optimized out>
        heads = <value optimized out>
        secs = <value optimized out>
        translation = <value optimized out>
        hda_opts = <value optimized out>
        opts = <value optimized out>
        olist = <value optimized out>
        optind = 52
        optarg = 0x7fffed7ff877 "chardev=mon,mode=control"
        loadvm = <value optimized out>
        machine = <value optimized out>
        cpu_model = <value optimized out>
        fds = {-930537472, 32689}
        tb_size = <value optimized out>
        pid_file = <value optimized out>
        incoming = <value optimized out>
        fd = <value optimized out>
        pwd = <value optimized out>
        chroot_dir = <value optimized out>
        run_as = <value optimized out>
        env = <value optimized out>
        show_vnc_port = <value optimized out>
        defconfig = <value optimized out>
        defconfig_verbose = <value optimized out>
(gdb)

Expect result:
the qemu should not core dumped, just quit with prompt is ok, like:
qemu-kvm: -chardev socket,id=mon,path=/opt/mon.monitor: Failed to connect to socket: No such file or directory
chardev: opening backend "socket" failed

Additional info:
If listen by nc -lU /path/to/unix-socket first and boot guest with '-qmp unix:/opt/mon.monitor', it work well without any problem. 
# nc -Ul /opt/mon.monitor
{"QMP": {"version": {"qemu": {"micro": 1, "minor": 12, "major": 0}, "package": "(qemu-kvm-0.12.1.2)"}, "capabilities": []}}
{"execute":"qmp_capabilities"}
{"return": {}}

Best Regards,
sluo

Comment 3 Sibiao Luo 2013-07-22 05:13:10 UTC
Also met the same core dumped bt log when boot guest with -chardev but without backend, qemu will quit with prompt, bug also core dump. Does it the same issue ?
e.g:...-device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev id=channel0,host=localhost,port=12345,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=channel0,id=channel0,name=com.redhat.rhevm.vdsm

# /usr/libexec/qemu-kvm -S -M rhel6.5.0 -cpu SandyBridge -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1 -no-kvm-pit-reinjection -name sluo -uuid 43425b70-86e5-4664-bf2c-3b76699b8bec -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev id=channel0,host=localhost,port=12345,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=channel0,id=channel0,name=com.redhat.rhevm.vdsm -drive file=/home/RHEL6.5-20130712.n.0-6.5-Server-x64.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK1" -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-system-disk,id=system-disk,bootindex=1 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x5 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=2C:41:38:B6:40:21,bus=pci.0,addr=0x6,bootindex=2 -drive file=/home/my-data-disk.qcow2,if=none,id=drive-data-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK2" -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-data-disk,id=data-disk,bootindex=3 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice port=5931,disable-ticketing -monitor stdio
chardev: "channel0" missing backend
Segmentation fault (core dumped)

(gdb) bt
#0  0x00007f6981948a9e in quit_timers () at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1858
#1  0x00007f697f288da2 in exit () from /lib64/libc.so.6
#2  0x00007f698194d851 in select_vgahw (argc=56, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4563
#3  main (argc=56, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:5661
(gdb) bt full
#0  0x00007f6981948a9e in quit_timers () at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1858
No locals.
#1  0x00007f697f288da2 in exit () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007f698194d851 in select_vgahw (argc=56, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4563
        opts = 0x7fff214a0347 ""
#3  main (argc=56, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:5661
        popt = <value optimized out>
        gdbstub_dev = <value optimized out>
        i = <value optimized out>
        snapshot = <value optimized out>
        linux_boot = <value optimized out>
        initrd_filename = <value optimized out>
        kernel_filename = <value optimized out>
        kernel_cmdline = <value optimized out>
        boot_devices = "cad", '\000' <repeats 29 times>
        ds = <value optimized out>
        dcl = <value optimized out>
        cyls = <value optimized out>
        heads = <value optimized out>
        secs = <value optimized out>
        translation = <value optimized out>
        hda_opts = <value optimized out>
        opts = <value optimized out>
        olist = <value optimized out>
        optind = 56
        optarg = 0x7fff214a088a "stdio"
        loadvm = <value optimized out>
        machine = <value optimized out>
        cpu_model = <value optimized out>
        fds = {-2121469952, 32617}
        tb_size = <value optimized out>
        pid_file = <value optimized out>
        incoming = <value optimized out>
        fd = <value optimized out>
        pwd = <value optimized out>
        chroot_dir = <value optimized out>
        run_as = <value optimized out>
        env = <value optimized out>
        show_vnc_port = <value optimized out>
        defconfig = <value optimized out>
        defconfig_verbose = <value optimized out>
(gdb)

Comment 4 Amit Shah 2013-07-22 13:05:11 UTC
Please test after chardev hotplug series is merged (bug 676568)

Comment 5 Sibiao Luo 2013-07-23 10:16:28 UTC
(In reply to Amit Shah from comment #4)
> Please test after chardev hotplug series is merged (bug 676568)
Ok, and i summarize this issue with three cases as following,
- comment #0:
 QEMU core dumped when attach a virtio serial device with "file" backend chardev
- comment #2:
 when boot up the guest with an chardev(unix socket, opened) and qmp monitor if there is no listen by nc -lU /path/to/unix-socket first.
- comment #3:
 Qemu core dumped when boot guest with -chardev but without backend.

Best Regards,
sluo

Comment 6 Sibiao Luo 2013-07-23 10:24:36 UTC
Also met it when two qemu process with the qmp using the same port.
# /usr/libexec/qemu-kvm -M rhel6.5.0 -cpu SandyBridge -enable-kvm -qmp tcp:0:4444,server,nowait -vnc :1 -monitor stdio
QEMU 0.12.1 monitor - type 'help' for more information
(qemu)

# /usr/libexec/qemu-kvm -M rhel6.5.0 -cpu SandyBridge -enable-kvm -qmp tcp:0:4444,server,nowait -vnc :2 -monitor stdio
inet_listen_opts: bind(ipv4,0.0.0.0,4444): Address already in use
inet_listen_opts: FAILED
chardev: opening backend "socket" failed
Segmentation fault (core dumped)

Best Regards,
sluo

Comment 7 Amit Shah 2013-07-29 07:28:50 UTC
Still reproduces after the chardev rework.  This doesn't reproduce upstream.  I spent a bit of time checking where things could be going wrong, but it's not immediately obvious.  Since this isn't a crash during a running instance of qemu, and libvirt won't see such crashes due to carefully selecting its chardevs, I suggest we don't pursue the fix for RHEL6.

Comment 9 Ademar Reis 2013-08-01 23:34:38 UTC
*** Bug 989306 has been marked as a duplicate of this bug. ***

Comment 10 Ademar Reis 2013-08-01 23:38:24 UTC
(In reply to Amit Shah from comment #7)
> Still reproduces after the chardev rework.  This doesn't reproduce upstream.
> I spent a bit of time checking where things could be going wrong, but it's
> not immediately obvious.  Since this isn't a crash during a running instance
> of qemu, and libvirt won't see such crashes due to carefully selecting its
> chardevs, I suggest we don't pursue the fix for RHEL6.

I agree. This is a corner case, easy to workaround and shouldn't affect customers. If one ever reports something, we'll reopen it. WONTFIX for RHEL6, works upstream/RHEL7.