Bug 2014139

Summary: [nodedev][vhba] virtnodedevd daemon crashed when creating a vhba device
Product: Red Hat Enterprise Linux 9 Reporter: yisun
Component: libvirtAssignee: Peter Krempa <pkrempa>
libvirt sub component: General QA Contact: yisun
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: jdenemar, jsuchane, lmen, pkrempa, smitterl, virt-maint, yisun, zhetang
Version: 9.0Keywords: Automation, Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-7.9.0-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-17 12:45:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version: 7.9.0
Embargoed:

Description yisun 2021-10-14 13:53:30 UTC
versions:
libvirt-7.8.0-1.el9.x86_64

how reproducible:
100%

steps:
1. prepare a host with hba card connecting to san storage, here we have scsi12 as the connected storage
[root@dell-per730-58 ~]# virsh nodedev-dumpxml scsi_host12
<device>
  <name>scsi_host12</name>
  <path>/sys/devices/pci0000:00/0000:00:03.2/0000:06:00.1/host12</path>
  <parent>pci_0000_06_00_1</parent>
  <capability type='scsi_host'>
    <host>12</host>
    <unique_id>12</unique_id>
    <capability type='fc_host'>
      <wwnn>2000f4e9d4eb02c9</wwnn>
      <wwpn>2001f4e9d4eb02c9</wwpn>
      <fabric_wwn>2001547feebe79c1</fabric_wwn>
    </capability>
    <capability type='vport_ops'>
      <max_vports>254</max_vports>
      <vports>0</vports>
    </capability>
  </capability>
</device>

2. prepare a device xml to generate a vhba device
[root@dell-per730-58 ~]# cat vhba.xml 
<device><parent>scsi_host12</parent><capability type='scsi_host'><capability type='fc_host'></capability></capability></device>

3. try to create the vhba 
[root@dell-per730-58 ~]# virsh nodedev-create vhba.xml 
error: Failed to create node device from vhba.xml
error: End of file while reading data: Input/output error

4. crash happens to /usr/sbin/virtnodedevd daemon
gdb backtrace info as follow for the segmentation fault:

...
(gdb) c
Continuing.

Thread 3 "rpc-worker" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff810116640 (LWP 122668)]
0x00007ff80c0ffacf in nodeDeviceDefPostParse (def=0x0, opaque=0x0) at ../src/node_device/node_device_driver.c:1906
1906	    for (caps = def->caps; caps != NULL; caps = caps->next) {

(gdb) t a a bt

Thread 19 (Thread 0x7ff7caffd640 (LWP 122685) "udev-event"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad645720) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad645720, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad645720, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad6456b8, cond=0x5634ad6456f8) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad6456f8, mutex=mutex@entry=0x5634ad6456b8) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad6456f8, m=m@entry=0x5634ad6456b8) at ../src/util/virthread.c:156
#6  0x00007ff80c107344 in udevEventHandleThread (opaque=<optimized out>) at ../src/node_device/node_device_udev.c:1802
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 18 (Thread 0x7ff7cb7fe640 (LWP 122684) "gdbus"):
#0  0x00007ff81208197f in __GI___poll (fds=0x7ff7c4020e10, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007ff8124ad5ac in g_main_context_poll (priority=<optimized out>, n_fds=2, fds=0x7ff7c4020e10, timeout=<optimized out>, context=0x7ff7c401dcb0) at ../glib/gmain.c:4434
#2  g_main_context_iterate.constprop.0 (context=0x7ff7c401dcb0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4126
#3  0x00007ff812458563 in g_main_loop_run (loop=0x7ff7c401dda0) at ../glib/gmain.c:4329
#4  0x00007ff8122e45ea in gdbus_shared_thread_func (user_data=0x7ff7c401dc80) at ../gio/gdbusprivate.c:280
#5  0x00007ff812487662 in g_thread_proxy (data=0x7ff7c4019400) at ../glib/gthread.c:826
#6  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#7  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 17 (Thread 0x7ff7cbfff640 (LWP 122683) "gmain"):
#0  0x00007ff81208197f in __GI___poll (fds=0x7ff7c40067e0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007ff8124ad5ac in g_main_context_poll (priority=<optimized out>, n_fds=2, fds=0x7ff7c40067e0, timeout=<optimized out>, context=0x7ff7c400c560) at ../glib/gmain.c:4434
#2  g_main_context_iterate.constprop.0 (context=context@entry=0x7ff7c400c560, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4126
#3  0x00007ff8124566d3 in g_main_context_iteration (context=0x7ff7c400c560, may_block=may_block@entry=1) at ../glib/gmain.c:4196
#4  0x00007ff812456721 in glib_worker_main (data=<optimized out>) at ../glib/gmain.c:6089
#5  0x00007ff812487662 in g_thread_proxy (data=0x7ff7c4009400) at ../glib/gthread.c:826
#6  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#7  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 16 (Thread 0x7ff7ed7fa640 (LWP 122681) "rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad648100) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad648100, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad648100, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad6480b0, cond=0x5634ad6480d8) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad6480d8, mutex=mutex@entry=0x5634ad6480b0) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad6480d8, m=m@entry=0x5634ad6480b0) at ../src/util/virthread.c:156
#6  0x00007ff8126713f3 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 15 (Thread 0x7ff7edffb640 (LWP 122680) "rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad648100) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad648100, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad648100, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad6480b0, cond=0x5634ad6480d8) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad6480d8, mutex=mutex@entry=0x5634ad6480b0) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad6480d8, m=m@entry=0x5634ad6480b0) at ../src/util/virthread.c:156
#6  0x00007ff8126713f3 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 14 (Thread 0x7ff7ee7fc640 (LWP 122679) "rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad648100) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad648100, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad648100, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad6480b0, cond=0x5634ad6480d8) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad6480d8, mutex=mutex@entry=0x5634ad6480b0) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad6480d8, m=m@entry=0x5634ad6480b0) at ../src/util/virthread.c:156
#6  0x00007ff8126713f3 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 13 (Thread 0x7ff7eeffd640 (LWP 122678) "rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad648100) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad648100, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
--Type <RET> for more, q to quit, c to continue without paging--
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad648100, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad6480b0, cond=0x5634ad6480d8) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad6480d8, mutex=mutex@entry=0x5634ad6480b0) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad6480d8, m=m@entry=0x5634ad6480b0) at ../src/util/virthread.c:156
#6  0x00007ff8126713f3 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 12 (Thread 0x7ff7ef7fe640 (LWP 122677) "rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad648100) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad648100, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad648100, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad6480b0, cond=0x5634ad6480d8) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad6480d8, mutex=mutex@entry=0x5634ad6480b0) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad6480d8, m=m@entry=0x5634ad6480b0) at ../src/util/virthread.c:156
#6  0x00007ff8126713f3 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 11 (Thread 0x7ff7effff640 (LWP 122676) "prio-rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad63daf4) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad63daf4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad63daf4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad63da00, cond=0x5634ad63dac8) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad63dac8, mutex=mutex@entry=0x5634ad63da00) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad63dac8, m=m@entry=0x5634ad63da00) at ../src/util/virthread.c:156
#6  0x00007ff8126713a8 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 10 (Thread 0x7ff80c90f640 (LWP 122675) "prio-rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad63daf4) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad63daf4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad63daf4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad63da00, cond=0x5634ad63dac8) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad63dac8, mutex=mutex@entry=0x5634ad63da00) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad63dac8, m=m@entry=0x5634ad63da00) at ../src/util/virthread.c:156
#6  0x00007ff8126713a8 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7ff80d110640 (LWP 122674) "prio-rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad63daf4) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad63daf4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad63daf4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad63da00, cond=0x5634ad63dac8) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad63dac8, mutex=mutex@entry=0x5634ad63da00) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad63dac8, m=m@entry=0x5634ad63da00) at ../src/util/virthread.c:156
#6  0x00007ff8126713a8 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7ff80d911640 (LWP 122673) "prio-rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad63daf4) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad63daf4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad63daf4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad63da00, cond=0x5634ad63dac8) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad63dac8, mutex=mutex@entry=0x5634ad63da00) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad63dac8, m=m@entry=0x5634ad63da00) at ../src/util/virthread.c:156
#6  0x00007ff8126713a8 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7ff80e112640 (LWP 122672) "prio-rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad63daf4) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad63daf4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
--Type <RET> for more, q to quit, c to continue without paging--
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad63daf4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad63da00, cond=0x5634ad63dac8) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad63dac8, mutex=mutex@entry=0x5634ad63da00) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad63dac8, m=m@entry=0x5634ad63da00) at ../src/util/virthread.c:156
#6  0x00007ff8126713a8 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7ff80e913640 (LWP 122671) "rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad63da54) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad63da54, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad63da54, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad63da00, cond=0x5634ad63da28) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad63da28, mutex=mutex@entry=0x5634ad63da00) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad63da28, m=m@entry=0x5634ad63da00) at ../src/util/virthread.c:156
#6  0x00007ff8126713f3 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7ff80f114640 (LWP 122670) "rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad63da54) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad63da54, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad63da54, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad63da00, cond=0x5634ad63da28) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad63da28, mutex=mutex@entry=0x5634ad63da00) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad63da28, m=m@entry=0x5634ad63da00) at ../src/util/virthread.c:156
#6  0x00007ff8126713f3 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7ff80f915640 (LWP 122669) "rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad63da54) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad63da54, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad63da54, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad63da00, cond=0x5634ad63da28) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad63da28, mutex=mutex@entry=0x5634ad63da00) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad63da28, m=m@entry=0x5634ad63da00) at ../src/util/virthread.c:156
#6  0x00007ff8126713f3 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81


Thread 3 (Thread 0x7ff810116640 (LWP 122668) "rpc-worker"):
#0  0x00007ff80c0ffacf in nodeDeviceDefPostParse (def=0x0, opaque=0x0) at ../src/node_device/node_device_driver.c:1906
#1  0x00007ff81270cda9 in virNodeDeviceDefParse (str=<optimized out>, filename=0x0, create=<optimized out>, virt_type=0x7ff80c108de4 "nodedev", parserCallbacks=0x7ff7c4023c98, opaque=0x0) at ../src/conf/node_device_conf.c:2189
#2  0x00007ff80c102a5e in nodeDeviceCreateXML (conn=0x7ff7c4014170, xmlDesc=0x7ff800002950 "<device><parent>scsi_host12</parent><capability type='scsi_host'><capability type='fc_host'></capability></capability></device>\n\n", flags=<optimized out>) at ../src/node_device/node_device_driver.c:901
#3  0x00007ff81284a3d0 in virNodeDeviceCreateXML (conn=0x7ff7c4014170, xmlDesc=0x7ff800002950 "<device><parent>scsi_host12</parent><capability type='scsi_host'><capability type='fc_host'></capability></capability></device>\n\n", flags=0) at ../src/libvirt-nodedev.c:690
#4  0x00005634ad341e49 in remoteDispatchNodeDeviceCreateXML (server=<optimized out>, msg=0x5634ad654450, ret=0x7ff800002690, args=0x7ff800001530, rerr=0x7ff8101159a0, client=0x5634ad653050) at src/remote/remote_daemon_dispatch_stubs.h:15086
#5  remoteDispatchNodeDeviceCreateXMLHelper (server=<optimized out>, client=0x5634ad653050, msg=0x5634ad654450, rerr=0x7ff8101159a0, args=0x7ff800001530, ret=0x7ff800002690) at src/remote/remote_daemon_dispatch_stubs.h:15067
#6  0x00007ff812735256 in virNetServerProgramDispatchCall (msg=0x5634ad654450, client=0x5634ad653050, server=0x5634ad645080, prog=0x5634ad647010) at ../src/rpc/virnetserverprogram.c:428
#7  virNetServerProgramDispatch (prog=0x5634ad647010, server=0x5634ad645080, client=0x5634ad653050, msg=0x5634ad654450) at ../src/rpc/virnetserverprogram.c:302
#8  0x00007ff812738cc8 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x5634ad645080) at ../src/rpc/virnetserver.c:135
#9  virNetServerHandleJob (jobOpaque=0x5634ad624160, opaque=0x5634ad645080) at ../src/rpc/virnetserver.c:152
#10 0x00007ff812671322 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:159
#11 0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#12 0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#13 0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7ff810917640 (LWP 122667) "rpc-worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5634ad63da50) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5634ad63da50, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007ff8120067bf in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5634ad63da50, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139
#3  0x00007ff812008ed0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5634ad63da00, cond=0x5634ad63da28) at pthread_cond_wait.c:504
#4  ___pthread_cond_wait (cond=cond@entry=0x5634ad63da28, mutex=mutex@entry=0x5634ad63da00) at pthread_cond_wait.c:619
#5  0x00007ff81267118b in virCondWait (c=c@entry=0x5634ad63da28, m=m@entry=0x5634ad63da00) at ../src/util/virthread.c:156
#6  0x00007ff8126713f3 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:116
#7  0x00007ff812678369 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:241
#8  0x00007ff812009af7 in start_thread (arg=<optimized out>) at pthread_create.c:435
#9  0x00007ff81208e830 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
--Type <RET> for more, q to quit, c to continue without paging--

Thread 1 (Thread 0x7ff810beeac0 (LWP 122666) "virtnodedevd"):
#0  0x00007ff81208197f in __GI___poll (fds=0x5634ad64bf00, nfds=9, timeout=5000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007ff8124ad5ac in g_main_context_poll (priority=<optimized out>, n_fds=9, fds=0x5634ad64bf00, timeout=<optimized out>, context=0x5634ad649d90) at ../glib/gmain.c:4434
#2  g_main_context_iterate.constprop.0 (context=context@entry=0x5634ad649d90, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4126
#3  0x00007ff8124566d3 in g_main_context_iteration (context=0x5634ad649d90, context@entry=0x0, may_block=may_block@entry=1) at ../glib/gmain.c:4196
#4  0x00007ff81261c704 in virEventGLibRunOnce () at ../src/util/vireventglib.c:516
#5  0x00007ff812736d4d in virNetDaemonRun (dmn=0x5634ad644820) at ../src/rpc/virnetdaemon.c:850
#6  0x00005634ad31833d in main (argc=<optimized out>, argv=<optimized out>) at ../src/remote/remote_daemon.c:1213
(gdb) 
(gdb)

Comment 2 Peter Krempa 2021-10-18 07:58:09 UTC
The issue is that the new post-parse callbacks for node devices are called even when the XML parser fails. This is not related to split vs monolithic daemon though.

Comment 4 Peter Krempa 2021-10-18 15:43:00 UTC
commit 8eb08e0fc5fe42efd2b0a4226f2b71f183ba193b
Author: Peter Krempa <pkrempa>
Date:   Mon Oct 18 09:54:08 2021 +0200

    virNodeDeviceDefParse: Don't call post-parse callbacks with NULL def
    
    When parsing of the node device XML fails we'd still call the post-parse
    and validation callbacks which makes no sense. Additionally the
    callbacks were expecting a non-NULL pointer which leads to a crash.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2014139
    Fixes: d5ae634ba28
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Erik Skultety <eskultet>

Comment 5 Peter Krempa 2021-11-03 08:12:05 UTC
*** Bug 2019701 has been marked as a duplicate of this bug. ***

Comment 6 yisun 2021-11-11 06:13:52 UTC
Verified with:
libvirt-7.9.0-1.el9.x86_64

Covered by auto cases
No crash happened

Comment 10 errata-xmlrpc 2022-05-17 12:45:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: libvirt), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2390