Bug 1201255

Created attachment 1000941 [details]
File: backtrace

Created attachment 1000942 [details]
File: cgroup

Created attachment 1000943 [details]
File: core_backtrace

Created attachment 1000944 [details]
File: dso_list

Created attachment 1000945 [details]
File: environ

Created attachment 1000946 [details]
File: limits

Created attachment 1000947 [details]
File: maps

Created attachment 1000948 [details]
File: open_fds

Created attachment 1000949 [details]
File: proc_pid_status

Created attachment 1000950 [details]
File: var_log_messages

*** Bug 1210282 has been marked as a duplicate of this bug. ***

This looks like a problem in libvirt-glib

libvirt-gconfig-domain.c is missing its share of g_return_val_if_fail() on its public entry point, but calling gvir_config_domain_get_name(NULL); is invalid, Boxes should not do that.

Looking at the code, this seems like a Vala bug. I'll try to reproduce here with different vala versions. I hope we updated vala in rhel7.2 though. At least 197 bugs were fixed in vala since GNOME 3.8: https://bugzilla.gnome.org/buglist.cgi?bug_status=RESOLVED&chfieldfrom=2013-10-01&chfieldto=Now&classification=Core&list_id=34992&product=vala&query_format=advanced&resolution=FIXED

Also were there any warnings on console?

(In reply to Zeeshan Ali from comment #14)
> Looking at the code, this seems like a Vala bug. I'll try to reproduce here
> with different vala versions. I hope we updated vala in rhel7.2 though. At
> least 197 bugs were fixed in vala since GNOME 3.8:
> https://bugzilla.gnome.org/buglist.cgi?bug_status=RESOLVED&chfieldfrom=2013-
> 10-
> 01&chfieldto=Now&classification=Core&list_id=34992&product=vala&query_format=
> advanced&resolution=FIXED

we have vala 0.26.1

This is the best I can get:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6d7a2cc in gvir_config_object_get_node_content ()
   from /lib64/libvirt-gconfig-1.0.so.0
Missing separate debuginfos, use: debuginfo-install cairo-1.14.2-2.el7.x86_64 cairo-gobject-1.14.2-2.el7.x86_64 glib2-2.42.2-2.el7.kalev0.x86_64 libnl3-3.2.21-8.el7.x86_64 nss-3.18.0-6.el7.x86_64 nss-util-3.18.0-2.el7_1.x86_64 openssl-libs-1.0.1e-42.el7_1.6.x86_64
(gdb) t a a bt

Thread 7 (Thread 0x7fffc6df0700 (LWP 9989)):
Python Exception <type 'exceptions.RuntimeError'> Cannot locate object file for block.: 
#0  0x00007ffff28c00bd in poll#1  0x00007ffff2dfaa74 in g_main_context_iterate.isra.24 () at /lib64/libglib-2.0.so.0
#2  0x00007ffff2dfab9c in g_main_context_iteration () at /lib64/libglib-2.0.so.0
#3  0x00007fffc6df82bd in dconf_gdbus_worker_thread (user_data=0xc614c0)
    at dconf-gdbus-thread.c:82
#4  0x00007ffff2e204e5 in g_thread_proxy () at /lib64/libglib-2.0.so.0
#5  0x00007ffff2b9cdf5 in start_thread (arg=0x7fffc6df0700)
    at pthread_create.c:308
Python Exception <type 'exceptions.RuntimeError'> Cannot locate object file for block.: 
#6  0x00007ffff28ca70d in clone
Thread 6 (Thread 0x7fffc77fe700 (LWP 9986)):
Python Exception <type 'exceptions.RuntimeError'> Cannot locate object file for block.: 
#0  0x00007ffff28c00bd in poll#1  0x00007ffff2dfaa74 in g_main_context_iterate.isra.24 () at /lib64/libglib-2.0.so.0
#2  0x00007ffff2dfab9c in g_main_context_iteration () at /lib64/libglib-2.0.so.0
#3  0x00007ffff2dfabd9 in glib_worker_main () at /lib64/libglib-2.0.so.0
#4  0x00007ffff2e204e5 in g_thread_proxy () at /lib64/libglib-2.0.so.0
#5  0x00007ffff2b9cdf5 in start_thread (arg=0x7fffc77fe700)
    at pthread_create.c:308
Python Exception <type 'exceptions.RuntimeError'> Cannot locate object file for block.: 
#6  0x00007ffff28ca70d in clone
Thread 3 (Thread 0x7fffd540c700 (LWP 9983)):
Python Exception <type 'exceptions.RuntimeError'> Cannot locate object file for block.: 
#0  0x00007ffff28c4e89 in syscall#1  0x00007ffff2e3dd9b in g_cond_wait_until ()
    at /lib64/libglib-2.0.so.0
#2  0x00007ffff2dd01f1 in g_async_queue_pop_intern_unlocked ()
    at /lib64/libglib-2.0.so.0
#3  0x00007ffff2dd07ab in g_async_queue_timeout_pop () at /lib64/libglib-2.0.so.0
#4  0x00007ffff2e20f2c in g_thread_pool_thread_proxy ()
    at /lib64/libglib-2.0.so.0
#5  0x00007ffff2e204e5 in g_thread_proxy () at /lib64/libglib-2.0.so.0
#6  0x00007ffff2b9cdf5 in start_thread (arg=0x7fffd540c700)
    at pthread_create.c:308
Python Exception <type 'exceptions.RuntimeError'> Cannot locate object file for block.: 
#7  0x00007ffff28ca70d in clone
Thread 2 (Thread 0x7fffd6eea700 (LWP 9982)):
Python Exception <type 'exceptions.RuntimeError'> Cannot locate object file for block.: 
#0  0x00007ffff28c00bd in poll#1  0x00007ffff2dfaa74 in g_main_context_iterate.isra.24 () at /lib64/libglib-2.0.so.0
#2  0x00007ffff2dfadba in g_main_loop_run () at /lib64/libglib-2.0.so.0
#3  0x00007ffff3414366 in gdbus_shared_thread_func () at /lib64/libgio-2.0.so.0
#4  0x00007ffff2e204e5 in g_thread_proxy () at /lib64/libglib-2.0.so.0
#5  0x00007ffff2b9cdf5 in start_thread (arg=0x7fffd6eea700)
    at pthread_create.c:308
Python Exception <type 'exceptions.RuntimeError'> Cannot locate object file for block.: 
#6  0x00007ffff28ca70d in clone
Thread 1 (Thread 0x7ffff7f2fb00 (LWP 9978)):
#0  0x00007ffff6d7a2cc in gvir_config_object_get_node_content ()
    at /lib64/libvirt-gconfig-1.0.so.0
#1  0x0000000000486fd5 in ______lambda161__gasync_ready_callback (_data82_=0xce3760, _data82_=0xce3760, obj=<optimized out>, _result_=<optimized out>)
    at libvirt-system-importer.c:655
#2  0x0000000000486fd5 in ______lambda161__gasync_ready_callback (source_object=<optimized out>, res=<optimized out>, self=0xce3760)
    at libvirt-system-importer.c:672
---Type <return> to continue, or q <return> to quit---
#3  0x00007ffff33aaf47 in g_simple_async_result_complete ()
    at /lib64/libgio-2.0.so.0
#4  0x00007ffff33aafa9 in complete_in_idle_cb () at /lib64/libgio-2.0.so.0
#5  0x00007ffff2dfa79a in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#6  0x00007ffff2dfaae8 in g_main_context_iterate.isra.24 ()
    at /lib64/libglib-2.0.so.0
#7  0x00007ffff2dfab9c in g_main_context_iteration () at /lib64/libglib-2.0.so.0
#8  0x00007ffff33ddf24 in g_application_run () at /lib64/libgio-2.0.so.0
#9  0x000000000045c4b9 in _vala_main (args=0x7fffffffd478, args_length1=1)
    at main.c:551
#10 0x00007ffff27f5af5 in __libc_start_main (main=
    0x42e360 <main>, argc=1, ubp_av=0x7fffffffd478, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd468)
    at libc-start.c:274
#11 0x000000000042e395 in _start ()
(gdb)

(In reply to Zeeshan Ali from comment #15)
> Also were there any warnings on console?

and no (as you can see in the gdb) there is absolutely nothing on the stderr.

(In reply to Matěj Cepl from comment #20)
> (In reply to Zeeshan Ali from comment #15)
> > Also were there any warnings on console?
> 
> and no (as you can see in the gdb) there is absolutely nothing on the stderr.

Thanks. Could you please attach the xml config of VMs being imported so I can try to reproduce here. `virsh -c qemu:///system list --all` will get you list of all VMs and `virsh -c qemu:///system dumpxml VM` will get you the XML.

Hey, any updates on this ?

(In reply to Zeeshan Ali from comment #21)
> (In reply to Matěj Cepl from comment #20)
> > (In reply to Zeeshan Ali from comment #15)
> > > Also were there any warnings on console?
> > 
> > and no (as you can see in the gdb) there is absolutely nothing on the stderr.
> 
> Thanks. Could you please attach the xml config of VMs being imported so I
> can try to reproduce here. `virsh -c qemu:///system list --all` will get you
> list of all VMs and `virsh -c qemu:///system dumpxml VM` will get you the
> XML.

matej@mitmanek: ~$ sudo virsh -c qemu:///system list --all
[sudo] password for matej: 
 Id    Name                           State
----------------------------------------------------
 -     filemon                        shut off
 -     santiago                       shut off
 -     simon                          shut off

matej@mitmanek: ~$ sudo virsh -c qemu:///system dumpxml filemon
<domain type='kvm'>
  <name>filemon</name>
  <uuid>a5145705-cfa0-4691-b028-67426c1a68b6</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/rhel/filemon'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:8f:64:13'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

matej@mitmanek: ~$ sudo virsh -c qemu:///system dumpxml santiago
<domain type='kvm'>
  <name>santiago</name>
  <uuid>e944117a-77ec-4bd9-bbf3-fa8a1f8000fa</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>3</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Nehalem</model>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/mapper/rhel-santiago'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/Windows10_InsiderPreview_x32_EN-US_10074.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:b0:7a:28'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='8192' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

matej@mitmanek: ~$ sudo virsh -c qemu:///system dumpxml simon
<domain type='kvm'>
  <name>simon</name>
  <uuid>952a1e40-9609-4c2d-b9d6-2ddb5b7a1feb</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>core2duo</model>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/rhel/simon'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/Windows10_InsiderPreview_x32_EN-US_10074.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:7d:d0:ab'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

matej@mitmanek: ~$

(In reply to Matěj Cepl from comment #23)

Ah, not sure yet but if I were to guess, the issue is your devices being real devices instead of images.

>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='none' io='native'/>
>       <source dev='/dev/rhel/filemon'/>
>       <target dev='vda' bus='virtio'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
>     </disk>

>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='none' io='native'/>
>       <source dev='/dev/mapper/rhel-santiago'/>
>       <target dev='vda' bus='virtio'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
>     </disk>

>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='none' io='native'/>
>       <source dev='/dev/rhel/simon'/>
>       <target dev='hda' bus='ide'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>

This is the first time I see anyone using real device so I'm sure I didn't take this into account or especially testing when developing this feature. I should soon have a RHEL7.2 VM to reproduce these bugs into (finally!) so I should be able to reproduce soon enough..

Just to say that I can reproduce this now and I already got a fix for the crash itselfand should have a fix for the feature itself soon too.

Imported RHEL7.2 with disk on physical drive (not image)

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2253.html