1849516 – libvirtd crashed when do a blockcommit

Bug 1849516 - libvirtd crashed when do a blockcommit

Summary: libvirtd crashed when do a blockcommit

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux Advanced Virtualization
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	8.3
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	8.3
Assignee:	Peter Krempa
QA Contact:	yisun
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-06-22 05:50 UTC by yisun
Modified:	2020-11-17 17:50 UTC (History)
CC List:	7 users (show)
Fixed In Version:	libvirt-6.5.0-1.el8
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-11-17 17:49:16 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description yisun 2020-06-22 05:50:12 UTC

Description:
libvirtd crashed when do a blockcommit

Versions:
libvirt-6.4.0-1.module+el8.3.0+6881+88468c00.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64

How reproducible:
100%

Steps:
1. Do a blockcommit from active layer
[root@libvirt-rhel-8 ~]# virsh snapshot-create-as avocado-vt-vm1 snap1 --disk-only
Domain snapshot snap1 created

[root@libvirt-rhel-8 ~]# virsh blockcommit avocado-vt-vm1 vda --active --pivot
error: Disconnected from qemu:///system due to end of file
error: internal error: client socket is closed

2. Do a blockcommit from middle layer to another middle layer will hit the same crash
[root@libvirt-rhel-8 rpm]# virsh snapshot-create-as avocado-vt-vm1 snap2 --disk-only
Domain snapshot snap2 created
[root@libvirt-rhel-8 rpm]# virsh snapshot-create-as avocado-vt-vm1 snap3 --disk-only
Domain snapshot snap3 created
[root@libvirt-rhel-8 rpm]# virsh blockcommit avocado-vt-vm1 vda^C-active --pivot
[root@libvirt-rhel-8 rpm]# virsh dumpxml avocado-vt-vm1 | awk '/<disk/,/<\/disk/'
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.snap3' index='4'/>
      <backingStore type='file' index='3'>
        <format type='qcow2'/>
        <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.snap2'/>
        <backingStore type='file' index='2'>
          <format type='qcow2'/>
          <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.snap1'/>
          <backingStore type='file' index='1'>
            <format type='qcow2'/>
            <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2'/>
            <backingStore/>
          </backingStore>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
[root@libvirt-rhel-8 rpm]# virsh blockcommit avocado-vt-vm1 vda --top vda[3] --base vda[1]
error: Disconnected from qemu:///system due to end of file
error: End of file while reading data: Input/output error

3. Gdb backtrace as follow:
Thread 6 "rpc-worker" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fa01b7fe700 (LWP 70861)]
0x00007f9ff18ab020 in qemuDomainDiskBlockJobIsSupported () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
(gdb) t a a bt

Thread 18 (Thread 0x7f9fe217f700 (LWP 70930)):
#0  0x00007fa0344d2ca1 in poll () from /lib64/libc.so.6
#1  0x00007fa0352649b6 in g_main_context_iterate.isra () from /lib64/libglib-2.0.so.0
#2  0x00007fa035264d72 in g_main_loop_run () from /lib64/libglib-2.0.so.0
#3  0x00007fa0383c1fce in virEventThreadWorker () from /lib64/libvirt.so.0
#4  0x00007fa03528cd4a in g_thread_proxy () from /lib64/libglib-2.0.so.0
#5  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f9fcbfff700 (LWP 70923)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f9ff224325c in udevEventHandleThread () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f9fe2980700 (LWP 70871)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c54b in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f9fe3181700 (LWP 70870)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c54b in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f9ff0800700 (LWP 70869)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c54b in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f9ff1001700 (LWP 70868)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c54b in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

--Type <RET> for more, q to quit, c to continue without paging--
Thread 12 (Thread 0x7f9ff1802700 (LWP 70867)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c54b in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fa018ff9700 (LWP 70866)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c508 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fa0197fa700 (LWP 70865)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c508 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fa019ffb700 (LWP 70864)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c508 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fa01a7fc700 (LWP 70863)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c508 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fa01affd700 (LWP 70862)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c508 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fa01b7fe700 (LWP 70861)):
#0  0x00007f9ff18ab020 in qemuDomainDiskBlockJobIsSupported () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
--Type <RET> for more, q to quit, c to continue without paging--
#1  0x00007f9ff1918692 in qemuDomainBlockCommit () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#2  0x00007fa0385de618 in virDomainBlockCommit () from /lib64/libvirt.so.0
#3  0x000055abe9ac334e in remoteDispatchDomainBlockCommit (server=0x55abea8f5dd0, msg=0x55abea958820, args=0x7fa008008030, rerr=0x7fa01b7fd8f0, client=<optimized out>) at ./remote/remote_daemon_dispatch_stubs.h:3887
#4  remoteDispatchDomainBlockCommitHelper (server=0x55abea8f5dd0, client=<optimized out>, msg=0x55abea958820, rerr=0x7fa01b7fd8f0, args=0x7fa008008030, ret=0x0) at ./remote/remote_daemon_dispatch_stubs.h:3858
#5  0x00007fa0385020d9 in virNetServerProgramDispatch () from /lib64/libvirt.so.0
#6  0x00007fa0385072a6 in virNetServerHandleJob () from /lib64/libvirt.so.0
#7  0x00007fa03841c47f in virThreadPoolWorker () from /lib64/libvirt.so.0
#8  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#9  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#10 0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fa013fff700 (LWP 70860)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c54b in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fa01bfff700 (LWP 70859)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c54b in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fa020a6b700 (LWP 70858)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c54b in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fa02126c700 (LWP 70857)):
#0  0x00007fa034bce2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa03841b92a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007fa03841c54b in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007fa03841ba7b in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007fa034bc814a in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa0344ddf23 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fa038f46c00 (LWP 70855)):
#0  0x00007fa0344d2ca1 in poll () from /lib64/libc.so.6
#1  0x00007fa0352649b6 in g_main_context_iterate.isra () from /lib64/libglib-2.0.so.0
#2  0x00007fa035264ae0 in g_main_context_iteration () from /lib64/libglib-2.0.so.0
#3  0x00007fa0383c1d74 in virEventGLibRunOnce () from /lib64/libvirt.so.0
#4  0x00007fa038506aa5 in virNetDaemonRun () from /lib64/libvirt.so.0
#5  0x000055abe9ab221f in main (argc=<optimized out>, argv=<optimized out>) at ../../src/remote/remote_daemon.c:1196

Comment 2 Peter Krempa 2020-06-22 07:08:53 UTC

Fixed upstream:

commit f225f37a8a49cedb847488599cf30c08567ba279
Author: Peter Krempa <pkrempa>
Date:   Fri Jun 5 12:19:29 2020 +0200

    qemu: blockcommit: Fix placement of qemuDomainDiskBlockJobIsSupported
    
    Commit b50a8354f6d added call to qemuDomainDiskBlockJobIsSupported prior
    to filling the 'disk' variable resulting in a crash when attempting a
    block commit.
    
    https://gitlab.com/libvirt/libvirt/-/issues/31
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

Comment 6 yisun 2020-07-16 02:59:02 UTC

libvirt version:
libvirt-6.5.0-1.module+el8.3.0+7323+d54bb644.x86_64

PASSed in latest auto task:
https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/libvirt/view/RHEL-8.3%20x86_64/job/libvirt-RHEL-8.3-runtest-x86_64-function-block_job_commit_pull/7/testReport/rhel.virsh/blockcommit/normal_test_multiple_chain_file_disk_local_no_ga_notimeout_nobase_top_active_with_pivot/

Comment 9 errata-xmlrpc 2020-11-17 17:49:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5137

Note You need to log in before you can comment on or make changes to this bug.