Red Hat Bugzilla – Bug 996814
boot image with gluster native mode cant work with attach another device from local file system
Last modified: 2016-09-20 00:39:59 EDT
Description of problem: boot guest from glusterfs server with gluster native mode, and attach a device which file index to local file, qemu-kvm will quit. Version-Release number of selected component (if applicable): host: RHEL6.5-20130804.n.0 qemu-kvm-tools-0.12.1.2-2.385.el6.g11.x86_64 gpxe-roms-qemu-0.9.7-6.9.el6.noarch qemu-kvm-0.12.1.2-2.385.el6.g11.x86_64 qemu-img-0.12.1.2-2.385.el6.g11.x86_64 kernel-2.6.32-411.el6.x86_64 gluster server: RHS-2.1-20130806.n.2 glusterfs-server-3.4.0.17rhs-1.el6rhs.x86_64 How reproducible: 100% Steps to Reproduce: 1.create a image on gluster sever 2.Try boot up it with attach another device from local file system /usr/libexec/qemu-kvm \ -M pc \ -cpu SandyBridge \ -m 4G \ -smp 4,sockets=2,cores=2,threads=1,maxcpus=16 \ -enable-kvm \ -name win2012 \ -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \ -smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \ -k en-us \ -rtc base=localtime,clock=host,driftfix=slew \ -nodefaults \ -monitor stdio \ -qmp tcp:0:6666,server,nowait \ -boot menu=on \ -bios /usr/share/seabios/bios.bin \ -monitor unix:/tmp/monitor-unix,nowait,server \ -vga qxl \ -spice port=5900,disable-ticketing \ -netdev tap,id=hostnet0,vhost=on \ -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:39:13:2c \ -drive file=gluster://m1/vol/win2012r2-free-install.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \ -device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0 \ -drive file=/home/rhel6u5.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop \ -device virtio-scsi-pci,id=scsi0,addr=0x5 \ -device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk,bootindex=1 \ Actual results: qemu-kvm quit with: Program received signal SIGUSR2, User defined signal 2. [Switching to Thread 0x7fffeebf6700 (LWP 30942)] 0x00007ffff772798e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt #0 0x00007ffff772798e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff4e5cb8f in syncenv_task (proc=0x7ffff8718410) at syncop.c:306 #2 0x00007ffff4e60d10 in syncenv_processor (thdata=0x7ffff8718410) at syncop.c:384 #3 0x00007ffff77239d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffff517ca8d in clone () from /lib64/libc.so.6 Expected results: qemu-kvm works well Additional info: 1 if both gluster native mode works well, eg: -drive file=gluster://m1/vol/win2012r2-free-install.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \ -device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0 \ -drive file=gluster://m1/vol/rhel6u5.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop \ -device virtio-scsi-pci,id=scsi0,addr=0x5 \ -device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk,bootindex=1 \ 2 with ide device also hit this problem, eg: -drive file=gluster://m1/vol/win2012r2-free-install.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \ -device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0 \ -drive file=/home/win2012r2.iso,if=none,media=cdrom,id=drive-ide0,readonly=on,format=raw \ -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0,id=ide0,bootindex=1
RHEL7 cant hit this problem with the same command line and glusterfs server. The version of qemu-kvm on RHEL7: qemu-kvm-common-1.5.2-3.el7.x86_64 ipxe-roms-qemu-20130517-1.gitc4bce43.el7.noarch qemu-img-1.5.2-3.el7.x86_64 qemu-kvm-tools-1.5.2-3.el7.x86_64 qemu-kvm-1.5.2-3.el7.x86_64
Update: 1) This works fine. No SIGUSR2 is observed. gdb --args $QEMU -L /usr/share/qemu-kvm -nographic -vnc :11 -enable-kvm -m 1024 \ -netdev tap,id=hn0,vhost=on -device virtio-net-pci,netdev=hn0 \ -drive file=$IMG,if=none,id=os -device virtio-blk-pci,drive=os,bootindex=1,scsi=off \ -drive file=gluster://gluster-server/vol/rhel6u5_2.qcow2,if=none,id=gfs0,cache=none,aio=native, -device virtio-blk-pci,drive=gfs0,bootindex=0 \ 2) If I switch the order of the two disks, SIGUSR2 is observed sometimes. gdb --args $QEMU -L /usr/share/qemu-kvm -nographic -vnc :11 -enable-kvm -m 1024 \ -netdev tap,id=hn0,vhost=on -device virtio-net-pci,netdev=hn0 \ -drive file=gluster://gluster-server/vol/rhel6u5_2.qcow2,if=none,id=gfs0,cache=none,aio=native, -device virtio-blk-pci,drive=gfs0,bootindex=0 \ -drive file=$IMG,if=none,id=os -device virtio-blk-pci,drive=os,bootindex=1,scsi=off \ Program received signal SIGUSR2, User defined signal 2.
> > Program received signal SIGUSR2, User defined signal 2. in scenario 2, after poweroff guest, sometimes hit the problem as well. Unmounting pipe file systems: [ OK ] Unmounting file systems: [ OK ] init: Re-executing /sbin/init Halting system... ACPI: Preparing to enter system sleep state S5 Disabling non-boot CPUs ... Power down. [New Thread 0x7fff751f4700 (LWP 3824)] Program received signal SIGUSR2, User defined signal 2. [Switching to Thread 0x7fffeebf6700 (LWP 3799)] 0x00007ffff772798e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt #0 0x00007ffff772798e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff4e5cb8f in syncenv_task () from /usr/lib64/libglusterfs.so.0 #2 0x00007ffff4e60d10 in syncenv_processor () from /usr/lib64/libglusterfs.so.0 #3 0x00007ffff77239d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffff517ca8d in clone () from /lib64/libc.so.6
Program received signal SIGUSR2, User defined signal 2. [Switching to Thread 0x7fffeebf4700 (LWP 7691)] pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239 239 62: movq %rax, %r14 (gdb) bt #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239 #1 0x00007ffff4c56b8f in syncenv_task (proc=0x7ffff8704c20) at syncop.c:306 #2 0x00007ffff4c5ad10 in syncenv_processor (thdata=0x7ffff8704c20) at syncop.c:384 #3 0x00007ffff77219d1 in start_thread (arg=0x7fffeebf4700) at pthread_create.c:301 #4 0x00007ffff4f76a8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Created attachment 788321 [details] gluster.sigusr2
With patch in Comment6 applied on Gluster. The issue is gone. The problem is that Gluster thread does not block the SIGUSR2 which is used by qemu-kvm's posix-aio-compat.c code.
Created attachment 788333 [details] Block SIGUSR2 only This also works.
Created attachment 788353 [details] Block all signals This works.
QEMU can not eliminate signals completely. In order to make gluster work with QEMU, gluster threads need to block them. It's necessary for threads to carefully block signals that are not essential to them.
Reproduce this bug with glusterfs-3.4.0.19rhs-2.el6.x86_64 and qemu-kvm-0.12.1.2-2.411.el6.x86_64. Update glusterfs and qemu-kvm package and re-test 5 times not hit this problem. host: RHEL6.5-Snapshot-2.0 [root@m2 ~]# rpm -qa |grep qemu qemu-kvm-0.12.1.2-2.412.el6.x86_64 gpxe-roms-qemu-0.9.7-6.10.el6.noarch qemu-kvm-debuginfo-0.12.1.2-2.412.el6.x86_64 qemu-img-0.12.1.2-2.412.el6.x86_64 qemu-kvm-tools-0.12.1.2-2.412.el6.x86_64 [root@m2 ~]# rpm -qa |grep glusterfs glusterfs-3.4.0.34rhs-1.el6.x86_64 glusterfs-api-3.4.0.34rhs-1.el6.x86_64 glusterfs-libs-3.4.0.34rhs-1.el6.x86_64 rhs: RHS-2.1-20130830.n.0 [root@rhs brick1]# rpm -qa |grep glusterfs glusterfs-server-3.4.0.34rhs-1.el6rhs.x86_64 samba-glusterfs-3.6.9-160.3.el6rhs.x86_64 glusterfs-3.4.0.34rhs-1.el6rhs.x86_64 glusterfs-fuse-3.4.0.34rhs-1.el6rhs.x86_64 glusterfs-api-3.4.0.34rhs-1.el6rhs.x86_64 glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs.x86_64 glusterfs-libs-3.4.0.34rhs-1.el6rhs.x86_64 glusterfs-rdma-3.4.0.34rhs-1.el6rhs.x86_64 steps and command line refer to comment#0 and comment#3. Result: qemu-kvm run well, no crash no SIGUSR2. so this bug has been fixed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-1553.html