Bug 1558934
| Summary: | libvirtd crash sometimes when update hostdev interface with '--config' | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | yafu <yafu> |
| Component: | libvirt | Assignee: | Laine Stump <laine> |
| Status: | CLOSED ERRATA | QA Contact: | yalzhang <yalzhang> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.5 | CC: | dyuan, fjin, jdenemar, jinqi, jiyan, jsuchane, xuzhang, yalzhang |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-4.5.0-27.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-31 19:58:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The way to trigger this bug is to call device-update with an <interface> definition that has no <address> element (and thus no virDeviceInfo). I tried this with latest git master (5.1.0), which didn't crash at all, and with the official RHEL 7.6 build (libvirt-4.5.0-10.el7_6.4) which *did* crash on the very first attempt:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fc4d20e7700 (LWP 11210)]
virDomainDefCollectBootOrder (def=0x7fc4a00100d0, dev=0x7fc4d20e67a0, info=0x0,
data=0x7fc4a0011d60) at conf/domain_conf.c:5072
5072 if (info->bootIndex == 0)
(gdb) bt
#0 virDomainDefCollectBootOrder (def=0x7fc4a00100d0, dev=0x7fc4d20e67a0, info=0x0,
data=0x7fc4a0011d60) at conf/domain_conf.c:5072
#1 0x00007fc4e25734c1 in virDomainDeviceInfoIterateInternal (
def=def@entry=0x7fc4a00100d0,
cb=cb@entry=0x7fc4e25742d0 <virDomainDefCollectBootOrder>, all=all@entry=false,
opaque=opaque@entry=0x7fc4a0011d60) at conf/domain_conf.c:3759
#2 0x00007fc4e2583c2a in virDomainDeviceInfoIterate (def=def@entry=0x7fc4a00100d0,
cb=cb@entry=0x7fc4e25742d0 <virDomainDefCollectBootOrder>,
opaque=opaque@entry=0x7fc4a0011d60) at conf/domain_conf.c:3933
#3 0x00007fc4e2596400 in virDomainDefBootOrderPostParse (def=0x7fc4a00100d0)
at conf/domain_conf.c:5113
#4 virDomainDefPostParseCommon (data=0x7fc4d20e6840, def=0x7fc4a00100d0)
at conf/domain_conf.c:5201
#5 virDomainDefPostParse (def=def@entry=0x7fc4a00100d0,
caps=caps@entry=0x7fc4ac189550, parseFlags=2, xmlopt=xmlopt@entry=0x7fc4ac188930,
parseOpaque=parseOpaque@entry=0x0) at conf/domain_conf.c:5332
#6 0x00007fc4b60a5017 in qemuDomainUpdateDeviceConfig (xmlopt=<optimized out>,
parse_flags=<optimized out>, caps=0x7fc4ac189550, dev=0x7fc4a0002940,
vmdef=<optimized out>) at qemu/qemu_driver.c:8465
#7 qemuDomainUpdateDeviceFlags (dom=0x7fc4a0000af0, xml=<optimized out>, flags=2)
at qemu/qemu_driver.c:8678
virDomainDefCollectBootOrder doesn't check that info != NULL, but more important, virDomainDeviceInfoIterateInternal blindly calls its callback function regardless of whether or not info is NULL (even though a survey of callback functions sent to it shows that they (just like virDomainDefCollectBootOrder) do not check for non-NULL info either.
Actually, the trigger for this is doing a virsh update-device on any <interface type='hostdev'>. The problem is that each virDomainNetDef that is of type hostdev internally has a virHostdevDef, and in order to make hostdev setup and teardown easier, the hostdevs array of the domain has a pointer to that virHostdevDef (in addition to the virDOmainNetDef having an entry in the domain's nets array). This is taken care of properly when an entry is removed from the nets array, but when we are just replacing an existing entry in nets, the pointer in the hostdevs into the old NetDef is left dangling. The *proper* fix is to get rid of all that horrible abomination to avoid similar problems in the future (I can say that because I wrote it :-/), but in the short term we just need to determine the simplest method of updating (or potentially adding/removing) the pointer in the hostdevs array when the nets entry is updated. It's complicated a bit by the desire to not change the ordering of the devices in the nets array (if we didn't care about that, we could just remove the existing nets entry and add the updated entry, thus automatically taking advantage of the existing code that updates the hostdevs array during remove/add). Fix posted upstream: https://www.redhat.com/archives/libvir-list/2019-September/msg01141.html And now pushed upstream:
commit 7e490cdad6364c82b326d5d9251826c757e8f77b (HEAD -> master, origin/master, origin/HEAD)
Author: Laine Stump <laine>
Date: Wed Sep 25 12:42:51 2019 -0400
conf: utility function to update entry in def->nets array
Reproduce the bug on libvirt-4.5.0-26.el7.x86_64 with the step in comment 0. Update libvirt to libvirt-4.5.0-27.el7.x86_64, and test as below, the result is as expected: 1. Start a vm as below: # virsh dumpxml rhel | grep /interface -B9 ... <interface type='hostdev' managed='yes'> <mac address='52:54:00:a5:4c:8f'/> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x01' slot='0x10' function='0x7'/> </source> <model type='virtio'/> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </interface> 2. prepare interface xml as below without pci address: # cat interface.xml <interface type='hostdev' managed='yes'> <mac address='52:54:00:a5:4c:8f'/> <source> <address type='pci' domain='0x0000' bus='0x01' slot='0x10' function='0x7'/> </source> <model type='virtio'/> </interface> 3. as hostdev type interface do not support live update, update the device with --config: # pidof libvirtd 5359 # for i in {1..100} ; do virsh update-device rhel interface.xml --config ; done; Device updated successfully ... Device updated successfully # pidof libvirtd 5359 # grep -i error /var/log/libvirt/libvirtd.log ==> no outputs libvirt do not crash during update-device for 100 times, libvirtd do not crash. Verified with libvirt-4.5.0-32.virtcov.el7.x86_64 & qemu-kvm-rhev-2.12.0-43.el7.x86_64. The steps are the same as the steps in comment 10. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1094 |
Description of problem: libvirtd crash sometimes when update hostdev interface with '--config' Version-Release number of selected component (if applicable): libvirt-3.9.0-14.el7_5.2.x86_64 How reproducible: 20% Steps to Reproduce: 1.Prepare a hostdev interface xml: #cat interface.xml <interface type='hostdev' managed='yes'> <mac address='66:18:0b:c2:85:b1'/> <source> <address type='pci' domain='0x0000' bus='0x05' slot='0x10' function='0x5'/> </source> <model type='virtio'/> </interface> 2.Coldplug the hostdev interface to a guest: #virsh attach-device interface.xml --config Device attached successfully 3.Update the hostdev interface device with '--config': # for i in {1..10} ; do virsh update-device vm1 interface.xml --config ; done Device updated successfully Device updated successfully Device updated successfully Device updated successfully Device updated successfully Device updated successfully error: Disconnected from qemu:///system due to end of file error: Failed to update device from interface.xml error: End of file while reading data: Input/output error error: failed to connect to the hypervisor error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused error: failed to connect to the hypervisor error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused error: failed to connect to the hypervisor error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused Actual results: Libvirtd crash sometimes when update hostdev interface with '--config' Expected results: Libvirtd should not crash when update hostdev interface with '--config' Additional info: 1.The backtrace of the crashed libvirtd: Thread 17 (Thread 0x7f3533926700 (LWP 49816)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ffb0cb8, m=m@entry=0x561d2ffb0c90) at util/virthread.c:154 #2 0x00007f3580336373 in virThreadPoolWorker (opaque=opaque@entry=0x561d2ffaf840) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f3580ffa8c0 (LWP 49801)): #0 0x00007f357d45ae4d in poll () from /lib64/libc.so.6 #1 0x00007f35802d9456 in poll (__timeout=5000, __nfds=10, __fds=<optimized out>) at /usr/include/bits/poll2.h:46 #2 virEventPollRunOnce () at util/vireventpoll.c:641 #3 0x00007f35802d7f32 in virEventRunDefaultImpl () at util/virevent.c:327 #4 0x00007f35804562cd in virNetDaemonRun (dmn=0x561d2ff674f0) at rpc/virnetdaemon.c:837 #5 0x0000561d2f3650ae in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1494 Thread 15 (Thread 0x7f356deec700 (LWP 49807)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ff65938, m=m@entry=0x561d2ff65870) at util/virthread.c:154 #2 0x00007f358033630b in virThreadPoolWorker (opaque=opaque@entry=0x561d2ff598e0) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f356f6ef700 (LWP 49804)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ff65898, m=m@entry=0x561d2ff65870) at util/virthread.c:154 #2 0x00007f3580336373 in virThreadPoolWorker (opaque=opaque@entry=0x561d2ff65160) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f3535129700 (LWP 49812)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ffb0cb8, m=m@entry=0x561d2ffb0c90) at util/virthread.c:154 #2 0x00007f3580336373 in virThreadPoolWorker (opaque=opaque@entry=0x561d2ffb0910) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f356e6ed700 (LWP 49806)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ff65898, m=m@entry=0x561d2ff65870) at util/virthread.c:154 #2 0x00007f3580336373 in virThreadPoolWorker (opaque=opaque@entry=0x561d2ff599a0) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 11 (Thread 0x7f356fef0700 (LWP 49803)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ff65898, m=m@entry=0x561d2ff65870) at util/virthread.c:154 #2 0x00007f3580336373 in virThreadPoolWorker (opaque=opaque@entry=0x561d2ff64fb0) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f356eeee700 (LWP 49805)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ff65898, m=m@entry=0x561d2ff65870) at util/virthread.c:154 #2 0x00007f3580336373 in virThreadPoolWorker (opaque=opaque@entry=0x561d2ff59a60) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f356d6eb700 (LWP 49808)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ff65938, m=m@entry=0x561d2ff65870) at util/virthread.c:154 #2 0x00007f358033630b in virThreadPoolWorker (opaque=opaque@entry=0x561d2ff59820) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f352b926700 (LWP 49815)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ffb0cb8, m=m@entry=0x561d2ffb0c90) at util/virthread.c:154 #2 0x00007f3580336373 in virThreadPoolWorker (opaque=opaque@entry=0x561d2ffb0850) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f35677fe700 (LWP 49811)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ff65938, m=m@entry=0x561d2ff65870) at util/virthread.c:154 #2 0x00007f358033630b in virThreadPoolWorker (opaque=opaque@entry=0x561d2ff64fb0) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f356ceea700 (LWP 49809)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ff65938, m=m@entry=0x561d2ff65870) at util/virthread.c:154 #2 0x00007f358033630b in virThreadPoolWorker (opaque=opaque@entry=0x561d2ff599a0) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f3534127700 (LWP 49814)): ---Type <return> to continue, or q <return> to quit--- #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ffb0cb8, m=m@entry=0x561d2ffb0c90) at util/virthread.c:154 #2 0x00007f3580336373 in virThreadPoolWorker (opaque=opaque@entry=0x561d2ffb0a60) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f3532711700 (LWP 49859)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x7f350c1077a0, m=m@entry=0x7f350c107760) at util/virthread.c:154 #2 0x00007f35361bb828 in udevEventHandleThread (opaque=<optimized out>) at node_device/node_device_udev.c:1729 #3 0x00007f3580335672 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f3567fff700 (LWP 49810)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ff65938, m=m@entry=0x561d2ff65870) at util/virthread.c:154 #2 0x00007f358033630b in virThreadPoolWorker (opaque=opaque@entry=0x561d2ff59a60) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f3534928700 (LWP 49813)): #0 0x00007f357d73f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35803358b6 in virCondWait (c=c@entry=0x561d2ffb0cb8, m=m@entry=0x561d2ffb0c90) at util/virthread.c:154 #2 0x00007f3580336373 in virThreadPoolWorker (opaque=opaque@entry=0x561d2ffb0790) at util/virthreadpool.c:124 #3 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f357d465aed in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f35706f1700 (LWP 49802)): #0 virDeviceInfoPCIAddressPresent (info=0x0) at conf/device_conf.h:208 #1 qemuDomainCollectPCIAddress (def=0x7f3550027530, device=0x7f35706f0640, info=0x0, opaque=0x7f3550003940) at qemu/qemu_domain_address.c:1208 #2 0x00007f358034dc11 in virDomainDeviceInfoIterateInternal (def=def@entry=0x7f3550027530, cb=cb@entry=0x7f3535634260 <qemuDomainCollectPCIAddress>, all=all@entry=false, opaque=opaque@entry=0x7f3550003940) at conf/domain_conf.c:3685 #3 0x00007f358035e7aa in virDomainDeviceInfoIterate (def=def@entry=0x7f3550027530, cb=cb@entry=0x7f3535634260 <qemuDomainCollectPCIAddress>, opaque=opaque@entry=0x7f3550003940) at conf/domain_conf.c:3851 #4 0x00007f353563418b in qemuDomainPCIAddressSetCreate (def=def@entry=0x7f3550027530, nbuses=nbuses@entry=18, dryRun=dryRun@entry=true) at qemu/qemu_domain_address.c:1370 #5 0x00007f3535636c47 in qemuDomainAssignPCIAddresses (obj=<optimized out>, driver=<optimized out>, qemuCaps=<optimized out>, def=<optimized out>) at qemu/qemu_domain_address.c:2311 #6 qemuDomainAssignAddresses (def=0x7f3550027530, qemuCaps=0x7f350c00ec90, driver=0x7f350c279cc0, obj=0x0, newDomain=<optimized out>) at qemu/qemu_domain_address.c:2846 #7 0x00007f35803702b5 in virDomainDefPostParseInternal (def=def@entry=0x7f3550027530, caps=caps@entry=0x7f350c2be6f0, parseFlags=parseFlags@entry=2, xmlopt=0x7f350c2c6320, parseOpaque=parseOpaque@entry=0x0, bootHash=bootHash@entry=0x0) at conf/domain_conf.c:5097 #8 0x00007f35803706c8 in virDomainDefPostParse (def=def@entry=0x7f3550027530, caps=caps@entry=0x7f350c2be6f0, parseFlags=parseFlags@entry=2, xmlopt=<optimized out>, parseOpaque=parseOpaque@entry=0x0) at conf/domain_conf.c:5127 #9 0x00007f35356ba9b7 in qemuDomainUpdateDeviceConfig (parse_flags=2, dev=0x7f35500029a0, dev=0x7f35500029a0, xmlopt=<optimized out>, caps=0x7f350c2be6f0, vmdef=0x7f3550027530) at qemu/qemu_driver.c:8391 #10 qemuDomainUpdateDeviceFlags (dom=0x7f3550000ae0, xml=<optimized out>, flags=2) at qemu/qemu_driver.c:8598 #11 0x00007f35803f15c7 in virDomainUpdateDeviceFlags (domain=domain@entry=0x7f3550000ae0, xml=0x7f3550000940 "<interface type='hostdev' managed='yes'>\n<mac address='66:18:0b:c2:85:b1'/>\n<source>\n<address type='pci' domain='0x0000' bus='0x04' slot='0x10' function='0x1'/>\n</source>\n<driver iommu='on' ats='on'/>"..., flags=2) at libvirt-domain.c:8338 ---Type <return> to continue, or q <return> to quit--- #12 0x0000561d2f38136e in remoteDispatchDomainUpdateDeviceFlags (server=0x561d2ff65610, msg=0x561d2ffb48c0, args=0x7f35500008c0, rerr=0x7f35706f0c10, client=<optimized out>) at remote_dispatch.h:12339 #13 remoteDispatchDomainUpdateDeviceFlagsHelper (server=0x561d2ff65610, client=<optimized out>, msg=0x561d2ffb48c0, rerr=0x7f35706f0c10, args=0x7f35500008c0, ret=0x7f3550000900) at remote_dispatch.h:12315 #14 0x00007f358045c132 in virNetServerProgramDispatchCall (msg=0x561d2ffb48c0, client=0x561d2ffb3ff0, server=0x561d2ff65610, prog=0x561d2ffb0890) at rpc/virnetserverprogram.c:437 #15 virNetServerProgramDispatch (prog=0x561d2ffb0890, server=server@entry=0x561d2ff65610, client=0x561d2ffb3ff0, msg=0x561d2ffb48c0) at rpc/virnetserverprogram.c:307 #16 0x0000561d2f3a1c7d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x561d2ff65610) at rpc/virnetserver.c:148 #17 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x561d2ff65610) at rpc/virnetserver.c:169 #18 0x00007f35803362c1 in virThreadPoolWorker (opaque=opaque@entry=0x561d2ff652a0) at util/virthreadpool.c:167 #19 0x00007f3580335648 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #20 0x00007f357d73bdd5 in start_thread () from /lib64/libpthread.so.0 #21 0x00007f357d465aed in clone () from /lib64/libc.so.6