Bug 1598436

Summary: AttachDeviceFlags failed, killed by SIGSEGV (core dumped)
Product: [Community] Virtualization Tools Reporter: amadev <m>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED NOTABUG QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: laine, libvirt-maint, m, rbalakri
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-10 12:21:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirtd.log
none
strace
none
nova-compute.log none

Description amadev 2018-07-05 13:10:26 UTC 
Created attachment 1456763 [details]
libvirtd.log

Description of problem:

I'm trying to attach network port with openstack.

root@cfg01:/etc/libvirt# ps aux| grep libvirt
root      8423  3.0  0.1 1593060 28288 ?       Ssl  12:08   0:00 /usr/sbin/libvirtd -l
root      8561  0.0  0.0  13056   912 pts/1    S+   12:08   0:00 grep --color=auto libvirt
libvirt+ 19858  4.7  0.5 1491880 122492 ?      Sl   08:22  10:45 qemu-system-x86_64 -enable-kvm -name gue
libvirt+ 20400  0.0  0.0  49992  2568 ?        S    08:23   0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/l
root@cfg01:/etc/libvirt# pid=8423
root@cfg01:/etc/libvirt# strace -p $pid -T -f -o /tmp/strace &
[1] 8566
root@cfg01:/etc/libvirt# strace: Process 8423 attached with 17 threads
root@cfg01:/etc/libvirt#
root@cfg01:/etc/libvirt# nova interface-attach --port-id $p $s
strace: Process 8817 attached
strace: Process 8818 attached
strace: Process 8819 attached
strace: Process 8820 attached
ERROR (ClientException): Failed to attach network adapter device to 9f4f241b-8b91-4d0b-a2c6-20632d73441e (HTTP 500) (Request-ID: req-46a30902-4e18-48ea-9d69-b69332dddcf2)
[1]+  Done                    strace -p $pid -T -f -o /tmp/strace
root@cfg01:/etc/libvirt# ps aux| grep libvirt
root      8836  0.4  0.1 1593032 28284 ?       Ssl  12:09   0:00 /usr/sbin/libvirtd -l
root      9637  0.0  0.0  13056   940 pts/1    S+   12:13   0:00 grep --color=auto libvirt
libvirt+ 19858  4.7  0.5 1491880 122492 ?      Sl   08:22  10:57 qemu-system-x86_64 -enable-kvm -name gue
libvirt+ 20400  0.0  0.0  49992  2568 ?        S    08:23   0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/l


Version-Release number of selected component (if applicable):

root@cfg01:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial
root@cfg01:~# uname -a
Linux cfg01 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@cfg01:~# 
root@cfg01:~# 
root@cfg01:~# virsh -V
Virsh command line tool of libvirt 4.0.0
See web site at https://libvirt.org/

Compiled with support for:
 Hypervisors: QEMU/KVM LXC UML Xen LibXL OpenVZ VMware VirtualBox ESX Test
 Networking: Remote Network Bridging Interface netcf Nwfilter VirtualPort
 Storage: Dir Disk Filesystem SCSI Multipath iSCSI LVM RBD Sheepdog Gluster ZFS
 Miscellaneous: Daemon Nodedev AppArmor Secrets Debug DTrace Readline
root@cfg01:~# 
root@cfg01:~# apt policy nova-compute
nova-compute:
  Installed: 2:17.0.4-2~u16.04+mcp33
  Candidate: 2:17.0.4-2~u16.04+mcp33
  Version table:
 *** 2:17.0.4-2~u16.04+mcp33 1100
       1100 http://mirror.mirantis.com/testing/openstack-queens/xenial xenial/main amd64 Packages
        100 /var/lib/dpkg/status
     2:13.1.4-0ubuntu4.2 500
        500 http://mcp-oscore.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     2:13.0.0-0ubuntu2 500
        500 http://mcp-oscore.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
Comment 1 amadev 2018-07-05 13:11:00 UTC 
Created attachment 1456765 [details]
strace
Comment 2 amadev 2018-07-05 13:12:07 UTC 
Created attachment 1456766 [details]
nova-compute.log
Comment 3 amadev 2018-07-05 13:18:02 UTC 
I see the error "libvirtError: End of file while reading data: Input/output error", but neither from libvirtd log nor strace log I could not find out 
what file libvirt is reading and why it is failed.
Comment 4 Laine Stump 2018-07-05 19:20:35 UTC 
(In reply to amadev from comment #3)
> I see the error "libvirtError: End of file while reading data: Input/output
> error", but neither from libvirtd log nor strace log I could not find out 
> what file libvirt is reading and why it is failed.

Usually that just means that the qemu process has terminated while libvirt is still waiting for input on its QMP socket.

A couple of comments:

1) we're 5 months beyond libvirt-4.0.0 upstream, so if you're wanting upstream to look into this bug, you should try to install the latest upstream libvirt (4.5.0 was just released) and see if the prolem persists.

2) since you're comfortable with strace, maybe you could attach gdb to the libvirtd process (after installing debuginfo, however that's done in Ubuntu - the following exercise is pointless without full debuginfo for all the files in libvirtd), perform the operation that triggers the segv, then when gdb breaks, enter the command "bt" and attach the results to this BZ. That output would be much more helpful than the strace output or libvirt debug logs, since it will show the exact line where the segv occurred.
Comment 5 amadev 2018-07-10 12:21:37 UTC 
Laine,

1) Yes, update helped. Even more, I compiled libvirt from git sources v4.0.0 and problem disappeared. Probably it's related to some packaging issue. Sorry for 
distraction.

2) I tried gdb but it showed "No stack" on those binaries.