Bug 986601 - [abrt] nbdkit-1.1.2-2.fc19: recv_request_send_reply: Process /usr/sbin/nbdkit was killed by signal 6 (SIGABRT)
[abrt] nbdkit-1.1.2-2.fc19: recv_request_send_reply: Process /usr/sbin/nbdkit...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: nbdkit (Show other bugs)
19
x86_64 Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Richard W.M. Jones
Fedora Extras Quality Assurance
abrt_hash:d1336357808df833fdd87955072...
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-20 17:33 EDT by Michael Scherer
Modified: 2013-08-01 23:43 EDT (History)
1 user (show)

See Also:
Fixed In Version: nbdkit-1.1.2-3.fc18
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-01 23:27:20 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
File: backtrace (7.35 KB, text/plain)
2013-07-20 17:34 EDT, Michael Scherer
no flags Details
File: cgroup (140 bytes, text/plain)
2013-07-20 17:34 EDT, Michael Scherer
no flags Details
File: core_backtrace (953 bytes, text/plain)
2013-07-20 17:34 EDT, Michael Scherer
no flags Details
File: dso_list (607 bytes, text/plain)
2013-07-20 17:34 EDT, Michael Scherer
no flags Details
File: environ (2.99 KB, text/plain)
2013-07-20 17:34 EDT, Michael Scherer
no flags Details
File: limits (1.29 KB, text/plain)
2013-07-20 17:34 EDT, Michael Scherer
no flags Details
File: maps (3.41 KB, text/plain)
2013-07-20 17:34 EDT, Michael Scherer
no flags Details
File: open_fds (246 bytes, text/plain)
2013-07-20 17:34 EDT, Michael Scherer
no flags Details
File: proc_pid_status (932 bytes, text/plain)
2013-07-20 17:34 EDT, Michael Scherer
no flags Details
File: var_log_messages (1.07 KB, text/plain)
2013-07-20 17:34 EDT, Michael Scherer
no flags Details
fix storage of sockaddr (1.30 KB, patch)
2013-07-21 03:54 EDT, Michael Scherer
no flags Details | Diff

  None (edit)
Description Michael Scherer 2013-07-20 17:33:58 EDT
Description of problem:
tried to test the update. It may be caused by inconsistant state on libvirt or guestfish side.

i started as non root : 
$ nbdkit -f  /usr/lib64/nbdkit/plugins/nbdkit-example3-plugin.so

then :
guestfish --ro -a nbd://localhost
><fs> run
libguestfs: error: could not create appliance through libvirt: Unable to read from monitor: Connection reset by peer [code=38 domain=10]


But i did some test, involving ctrl C on both sides.

Version-Release number of selected component:
nbdkit-1.1.2-2.fc19

Additional info:
reporter:       libreport-2.1.5
backtrace_rating: 4
cmdline:        nbdkit -f /usr/lib64/nbdkit/plugins/nbdkit-example1-plugin.so
crash_function: recv_request_send_reply
executable:     /usr/sbin/nbdkit
kernel:         3.9.9-302.fc19.x86_64
runlevel:       N 5
uid:            500

Truncated backtrace:
Thread no. 1 (4 frames)
 #5 recv_request_send_reply at connections.c:429
 #6 _handle_single_connection at connections.c:82
 #7 handle_single_connection at connections.c:103
 #8 start_thread at sockets.c:220
Comment 1 Michael Scherer 2013-07-20 17:34:02 EDT
Created attachment 776274 [details]
File: backtrace
Comment 2 Michael Scherer 2013-07-20 17:34:04 EDT
Created attachment 776275 [details]
File: cgroup
Comment 3 Michael Scherer 2013-07-20 17:34:07 EDT
Created attachment 776276 [details]
File: core_backtrace
Comment 4 Michael Scherer 2013-07-20 17:34:10 EDT
Created attachment 776277 [details]
File: dso_list
Comment 5 Michael Scherer 2013-07-20 17:34:13 EDT
Created attachment 776278 [details]
File: environ
Comment 6 Michael Scherer 2013-07-20 17:34:15 EDT
Created attachment 776279 [details]
File: limits
Comment 7 Michael Scherer 2013-07-20 17:34:18 EDT
Created attachment 776280 [details]
File: maps
Comment 8 Michael Scherer 2013-07-20 17:34:23 EDT
Created attachment 776281 [details]
File: open_fds
Comment 9 Michael Scherer 2013-07-20 17:34:27 EDT
Created attachment 776282 [details]
File: proc_pid_status
Comment 10 Michael Scherer 2013-07-20 17:34:30 EDT
Created attachment 776283 [details]
File: var_log_messages
Comment 11 Richard W.M. Jones 2013-07-20 17:54:52 EDT
(In reply to Michael Scherer from comment #0)
> Description of problem:
> tried to test the update. It may be caused by inconsistant state on libvirt
> or guestfish side.
> 
> i started as non root : 
> $ nbdkit -f  /usr/lib64/nbdkit/plugins/nbdkit-example3-plugin.so
> 
> then :
> guestfish --ro -a nbd://localhost
> ><fs> run
> libguestfs: error: could not create appliance through libvirt: Unable to
> read from monitor: Connection reset by peer [code=38 domain=10]

If you use the guestfish -v option you should get a lot more
detail about what's really going on here.  This is possibly
a separate bug in libvirt.

But in any case, nbdkit shouldn't segfault even if you ^C it.
Comment 12 Michael Scherer 2013-07-20 18:20:18 EDT
Indeed, there is this log.

><fs> run
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfseJV2h8
libguestfs: launch: umask=0002
libguestfs: launch: euid=500
libguestfs: libvirt version = 1000005 (1.0.5)
libguestfs: [00000ms] connect to libvirt
libguestfs: opening libvirt handle: URI = NULL, auth = virConnectAuthPtrDefault, flags = 0
libguestfs: successfully opened libvirt handle: conn = 0x7fc709a859b0
libguestfs: [02077ms] get libvirt capabilities
libguestfs: [02083ms] parsing capabilities XML
libguestfs: [02084ms] build appliance
libguestfs: command: run: supermin-helper
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ -f checksum
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ x86_64
supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = (null), initrd = (null), appliance = (null)
supermin helper [00000ms] inputs[0] = /usr/lib64/guestfs/supermin.d
checking modpath /lib/modules/3.9.9-302.fc19.x86_64 is a directory
picked vmlinuz-3.9.9-302.fc19.x86_64 because modpath /lib/modules/3.9.9-302.fc19.x86_64 exists
checking modpath /lib/modules/3.9.8-300.fc19.x86_64 is a directory
picked vmlinuz-3.9.8-300.fc19.x86_64 because modpath /lib/modules/3.9.8-300.fc19.x86_64 exists
checking modpath /lib/modules/3.9.9-301.fc19.x86_64 is a directory
picked vmlinuz-3.9.9-301.fc19.x86_64 because modpath /lib/modules/3.9.9-301.fc19.x86_64 exists
supermin helper [00000ms] finished creating kernel
supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d
supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/base.img
supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/daemon.img
supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/hostfiles
supermin helper [00052ms] visiting /usr/lib64/guestfs/supermin.d/init.img
supermin helper [00052ms] visiting /usr/lib64/guestfs/supermin.d/udev-rules.img
supermin helper [00052ms] adding kernel modules
supermin helper [00088ms] finished creating appliance
libguestfs: checksum of existing appliance: d2ca7ac6cb24dfbbcacc5a4560fc754808221f87eae8c45e4816a8fa7a8365a4
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -b /var/tmp/.guestfs-500/root.25808
libguestfs: command: run: \ -o backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfseJV2h8/snapshot1
Formatting '/tmp/libguestfseJV2h8/snapshot1', fmt=qcow2 size=4294967296 backing_file='/var/tmp/.guestfs-500/root.25808' backing_fmt='raw' encryption=off cluster_size=65536 lazy_refcounts=off 
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -b nbd:localhost:10809
libguestfs: command: run: \ /tmp/libguestfseJV2h8/snapshot2
Formatting '/tmp/libguestfseJV2h8/snapshot2', fmt=qcow2 size=104857600 backing_file='nbd:localhost:10809' encryption=off cluster_size=65536 lazy_refcounts=off 
libguestfs: [02259ms] create libvirt XML
libguestfs: libvirt XML:\n<?xml version="1.0"?>\n<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">\n  <name>guestfs-4p3mepfyp0avvxwo</name>\n  <memory unit="MiB">500</memory>\n  <currentMemory unit="MiB">500</currentMemory>\n  <vcpu>1</vcpu>\n  <clock offset="utc"/>\n  <os>\n    <type>hvm</type>\n    <kernel>/var/tmp/.guestfs-500/kernel.25808</kernel>\n    <initrd>/var/tmp/.guestfs-500/initrd.25808</initrd>\n    <cmdline>panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm</cmdline>\n  </os>\n  <on_reboot>destroy</on_reboot>\n  <devices>\n    <controller type="scsi" index="0" model="virtio-scsi"/>\n    <disk device="disk" type="file">\n      <source file="/tmp/libguestfseJV2h8/snapshot2"/>\n      <target dev="sda" bus="scsi"/>\n      <driver name="qemu" type="qcow2"/>\n      <address type="drive" controller="0" bus="0" target="0" unit="0"/>\n    </disk>\n    <disk type="file" device="disk">\n      <source file="/tmp/libguestfseJV2h8/snapshot1"/>\n      <target dev="sdb" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="1" unit="0"/>\n      <shareable/>\n    </disk>\n    <serial type="unix">\n      <source mode="connect" path="/tmp/libguestfseJV2h8/console.sock"/>\n      <target port="0"/>\n    </serial>\n    <channel type="unix">\n      <source mode="connect" path="/tmp/libguestfseJV2h8/guestfsd.sock"/>\n      <target type="virtio" name="org.libguestfs.channel.0"/>\n    </channel>\n  </devices>\n  <qemu:commandline>\n    <qemu:env name="TMPDIR" value="/var/tmp"/>\n  </qemu:commandline>\n</domain>\n
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -Z /var/tmp/.guestfs-500
libguestfs: drwxr-xr-x. misc misc staff_u:object_r:user_tmp_t:s0   .
libguestfs: drwxrwxrwt. root root system_u:object_r:tmp_t:s0       ..
libguestfs: -rwxr-xr-x. misc misc staff_u:object_r:user_tmp_t:s0   checksum
libguestfs: -rw-r--r--. misc misc system_u:object_r:virt_content_t:s0 initrd
libguestfs: -rw-r--r--. misc misc system_u:object_r:virt_content_t:s0 initrd.25808
libguestfs: -rw-r--r--. misc misc system_u:object_r:virt_content_t:s0 kernel
libguestfs: -rw-r--r--. misc misc system_u:object_r:virt_content_t:s0 kernel.25808
libguestfs: -rw-r--r--. misc misc system_u:object_r:virt_content_t:s0 root
libguestfs: -rw-r--r--. misc misc system_u:object_r:virt_content_t:s0 root.25808
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -Z /tmp/libguestfseJV2h8
libguestfs: drwxr-xr-x. misc misc staff_u:object_r:user_tmp_t:s0   .
libguestfs: drwxrwxrwt. root root system_u:object_r:tmp_t:s0       ..
libguestfs: srwxrwxr-x. misc misc staff_u:object_r:user_tmp_t:s0   console.sock
libguestfs: srwxrwxr-x. misc misc staff_u:object_r:user_tmp_t:s0   guestfsd.sock
libguestfs: -rw-r--r--. misc misc staff_u:object_r:user_tmp_t:s0   snapshot1
libguestfs: -rw-r--r--. misc misc staff_u:object_r:user_tmp_t:s0   snapshot2
libguestfs: -rwxrwxr-x. misc misc staff_u:object_r:user_tmp_t:s0   umask-check
libguestfs: [02265ms] launch libvirt guest
libguestfs: error: could not create appliance through libvirt: internal error process exited while connecting to monitor: nbd.c:nbd_receive_negotiate():L457: read failed
qemu-system-x86_64: -drive file=/tmp/libguestfseJV2h8/snapshot2,if=none,id=drive-scsi0-0-0-0,format=qcow2: could not open disk image /tmp/libguestfseJV2h8/snapshot2: Invalid argument
 [code=1 domain=10]

I suspected selinux ( as I am running in a enforcing login ), but after setenforce 0, this still break.
Comment 13 Michael Scherer 2013-07-20 19:05:01 EDT
in fact, just doing : 
$ nbdkit -f /usr/lib64/nbdkit/plugins/nbdkit-example3-plugin.so

and on another terminal :
$ cat /dev/random| nc localhost 10809

is sufficient to trigger the bug.
Comment 14 Michael Scherer 2013-07-20 19:18:50 EDT
So trying to run with vlagrind and from git, it doesn't crash. The only message is see is :
==4204== Thread 2:
==4204== Invalid write of size 2
==4204==    at 0x4A0A404: memcpy@@GLIBC_2.14 (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4204==  Address 0x4c3fef8 is 0 bytes after a block of size 40 alloc'd
==4204==    at 0x4A08121: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4204==    by 0x404F02: tls_new_server_thread (tls.c:89)
==4204==    by 0x404764: start_thread (sockets.c:216)
==4204==    by 0x3112E07C52: start_thread (pthread_create.c:308)
==4204==    by 0x3112AF513C: clone (clone.S:113)
==4204== 

Running from git without valgrind is still crashing.
Comment 15 Michael Scherer 2013-07-21 03:52:00 EDT
Discussing with a friend of mine during the night, he found out the problem right away, so I coded a patch based on what he suggested, and this fix the error  in valgrind and I cannot reproduce crash.

Here is a patch against latest git HEAD.
Comment 16 Michael Scherer 2013-07-21 03:54:34 EDT
Created attachment 776370 [details]
fix storage of sockaddr

Discussing with a friend of mine during the night, he found out the problem right away, so I coded a patch based on what he suggested, and this fix the error  in valgrind and I cannot reproduce crash.

Here is a patch against latest git HEAD. 

As he explained, we use sockaddr to store sockaddr_in ( which is ok, same size ), but not for sockaddr_in6, because it is bigger, so memcopy goes over the bound of calloc.
Comment 17 Richard W.M. Jones 2013-07-21 17:05:13 EDT
Oh dear, that's embarrassingly bad.  I've pushed your change
with some minor reformatting.
Comment 18 Richard W.M. Jones 2013-07-21 17:46:08 EDT
Thanks.  This is now fixed in Rawhide, F19 & F18 (updates-testing).
Comment 19 Fedora Update System 2013-07-21 17:55:32 EDT
nbdkit-1.1.2-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/nbdkit-1.1.2-3.fc18
Comment 20 Fedora Update System 2013-07-21 17:55:38 EDT
nbdkit-1.1.2-3.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/nbdkit-1.1.2-3.fc19
Comment 21 Fedora Update System 2013-07-22 21:06:14 EDT
Package nbdkit-1.1.2-3.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing nbdkit-1.1.2-3.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-13377/nbdkit-1.1.2-3.fc19
then log in and leave karma (feedback).
Comment 22 Fedora Update System 2013-08-01 23:27:20 EDT
nbdkit-1.1.2-3.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 23 Fedora Update System 2013-08-01 23:43:25 EDT
nbdkit-1.1.2-3.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.