RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 996814 - boot image with gluster native mode cant work with attach another device from local file system
Summary: boot image with gluster native mode cant work with attach another device from...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Stefan Hajnoczi
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 998832
Blocks: 1000882
TreeView+ depends on / blocked
 
Reported: 2013-08-14 03:17 UTC by mazhang
Modified: 2016-09-20 04:39 UTC (History)
13 users (show)

Fixed In Version: qemu-kvm-0.12.1.2-2.412.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 998832 (view as bug list)
Environment:
Last Closed: 2013-11-21 07:10:35 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
gluster.sigusr2 (3.95 KB, patch)
2013-08-20 06:08 UTC, Asias He 		no flags Details | Diff
Block SIGUSR2 only (3.75 KB, patch)
2013-08-20 06:22 UTC, Asias He 		no flags Details | Diff
Block all signals (3.52 KB, patch)
2013-08-20 07:58 UTC, Asias He 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2013:1553 0 normal SHIPPED_LIVE Important: qemu-kvm security, bug fix, and enhancement update 2013-11-20 21:40:29 UTC

Description mazhang 2013-08-14 03:17:55 UTC 
Description of problem:
boot guest from glusterfs server with gluster native mode, and attach a device which file index to local file, qemu-kvm will quit.

Version-Release number of selected component (if applicable):

host: RHEL6.5-20130804.n.0
qemu-kvm-tools-0.12.1.2-2.385.el6.g11.x86_64
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
qemu-kvm-0.12.1.2-2.385.el6.g11.x86_64
qemu-img-0.12.1.2-2.385.el6.g11.x86_64
kernel-2.6.32-411.el6.x86_64

gluster server: RHS-2.1-20130806.n.2
glusterfs-server-3.4.0.17rhs-1.el6rhs.x86_64


How reproducible:
100%

Steps to Reproduce:
1.create a image on gluster sever

2.Try boot up it with attach another device from local file system
/usr/libexec/qemu-kvm \
-M pc \
-cpu SandyBridge \
-m 4G \
-smp 4,sockets=2,cores=2,threads=1,maxcpus=16 \
-enable-kvm \
-name win2012 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-rtc base=localtime,clock=host,driftfix=slew \
-nodefaults \
-monitor stdio \
-qmp tcp:0:6666,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-monitor unix:/tmp/monitor-unix,nowait,server \
-vga qxl \
-spice port=5900,disable-ticketing \
-netdev tap,id=hostnet0,vhost=on \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:39:13:2c \
-drive file=gluster://m1/vol/win2012r2-free-install.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0 \
-drive file=/home/rhel6u5.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-scsi-pci,id=scsi0,addr=0x5 \
-device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk,bootindex=1 \



Actual results:
qemu-kvm quit with:
Program received signal SIGUSR2, User defined signal 2.
[Switching to Thread 0x7fffeebf6700 (LWP 30942)]
0x00007ffff772798e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff772798e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff4e5cb8f in syncenv_task (proc=0x7ffff8718410) at syncop.c:306
#2  0x00007ffff4e60d10 in syncenv_processor (thdata=0x7ffff8718410) at syncop.c:384
#3  0x00007ffff77239d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007ffff517ca8d in clone () from /lib64/libc.so.6

Expected results:
qemu-kvm works well

Additional info:
1 if both gluster native mode works well, eg:
-drive file=gluster://m1/vol/win2012r2-free-install.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0 \
-drive file=gluster://m1/vol/rhel6u5.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-scsi-pci,id=scsi0,addr=0x5 \
-device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk,bootindex=1 \

2 with ide device also hit this problem, eg:
-drive file=gluster://m1/vol/win2012r2-free-install.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0 \
-drive file=/home/win2012r2.iso,if=none,media=cdrom,id=drive-ide0,readonly=on,format=raw \
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0,id=ide0,bootindex=1
Comment 2 mazhang 2013-08-14 05:08:41 UTC 
RHEL7 cant hit this problem with the same command line and glusterfs server.
The version of qemu-kvm on RHEL7:
qemu-kvm-common-1.5.2-3.el7.x86_64
ipxe-roms-qemu-20130517-1.gitc4bce43.el7.noarch
qemu-img-1.5.2-3.el7.x86_64
qemu-kvm-tools-1.5.2-3.el7.x86_64
qemu-kvm-1.5.2-3.el7.x86_64
Comment 3 Asias He 2013-08-14 09:01:15 UTC 
Update: 

1) This works fine. No SIGUSR2 is observed.
gdb --args $QEMU -L /usr/share/qemu-kvm -nographic -vnc :11 -enable-kvm -m 1024 \
-netdev tap,id=hn0,vhost=on -device virtio-net-pci,netdev=hn0 \
-drive file=$IMG,if=none,id=os -device virtio-blk-pci,drive=os,bootindex=1,scsi=off \
-drive file=gluster://gluster-server/vol/rhel6u5_2.qcow2,if=none,id=gfs0,cache=none,aio=native, -device virtio-blk-pci,drive=gfs0,bootindex=0 \

2) If I switch the order of the two disks, SIGUSR2 is observed sometimes.
gdb --args $QEMU -L /usr/share/qemu-kvm -nographic -vnc :11 -enable-kvm -m 1024 \
-netdev tap,id=hn0,vhost=on -device virtio-net-pci,netdev=hn0 \
-drive file=gluster://gluster-server/vol/rhel6u5_2.qcow2,if=none,id=gfs0,cache=none,aio=native, -device virtio-blk-pci,drive=gfs0,bootindex=0 \
-drive file=$IMG,if=none,id=os -device virtio-blk-pci,drive=os,bootindex=1,scsi=off \


Program received signal SIGUSR2, User defined signal 2.
Comment 4 mazhang 2013-08-14 09:12:32 UTC 
> 
> Program received signal SIGUSR2, User defined signal 2.

in scenario 2, after poweroff guest, sometimes hit the problem as well.
Unmounting pipe file systems:  [  OK  ]
Unmounting file systems:  [  OK  ]
init: Re-executing /sbin/init
Halting system...
ACPI: Preparing to enter system sleep state S5
Disabling non-boot CPUs ...
Power down.
[New Thread 0x7fff751f4700 (LWP 3824)]

Program received signal SIGUSR2, User defined signal 2.
[Switching to Thread 0x7fffeebf6700 (LWP 3799)]
0x00007ffff772798e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff772798e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ffff4e5cb8f in syncenv_task () from /usr/lib64/libglusterfs.so.0
#2  0x00007ffff4e60d10 in syncenv_processor () from /usr/lib64/libglusterfs.so.0
#3  0x00007ffff77239d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007ffff517ca8d in clone () from /lib64/libc.so.6
Comment 5 mazhang 2013-08-15 06:24:46 UTC 
Program received signal SIGUSR2, User defined signal 2.
[Switching to Thread 0x7fffeebf4700 (LWP 7691)]
pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
239	62:	movq	%rax, %r14
(gdb) bt
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
#1  0x00007ffff4c56b8f in syncenv_task (proc=0x7ffff8704c20) at syncop.c:306
#2  0x00007ffff4c5ad10 in syncenv_processor (thdata=0x7ffff8704c20) at syncop.c:384
#3  0x00007ffff77219d1 in start_thread (arg=0x7fffeebf4700) at pthread_create.c:301
#4  0x00007ffff4f76a8d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Comment 6 Asias He 2013-08-20 06:08:16 UTC 
Created attachment 788321 [details]
gluster.sigusr2
Comment 7 Asias He 2013-08-20 06:12:17 UTC 
With patch in Comment6 applied on Gluster. The issue is gone.

The problem is that Gluster thread does not block the SIGUSR2 which is used by qemu-kvm's posix-aio-compat.c code.
Comment 8 Asias He 2013-08-20 06:22:31 UTC 
Created attachment 788333 [details]
Block SIGUSR2 only

This also works.
Comment 9 Asias He 2013-08-20 07:58:59 UTC 
Created attachment 788353 [details]
Block all signals

This works.
Comment 10 Asias He 2013-08-20 08:06:15 UTC 
QEMU can not eliminate signals completely. In order to make gluster work with QEMU, gluster threads need to block them. It's necessary for threads to carefully block signals that are not essential to them.
Comment 13 mazhang 2013-10-12 08:48:20 UTC 
Reproduce this bug with glusterfs-3.4.0.19rhs-2.el6.x86_64 and qemu-kvm-0.12.1.2-2.411.el6.x86_64.

Update glusterfs and qemu-kvm package and re-test 5 times not hit this problem.

host:
RHEL6.5-Snapshot-2.0
[root@m2 ~]# rpm -qa |grep qemu
qemu-kvm-0.12.1.2-2.412.el6.x86_64
gpxe-roms-qemu-0.9.7-6.10.el6.noarch
qemu-kvm-debuginfo-0.12.1.2-2.412.el6.x86_64
qemu-img-0.12.1.2-2.412.el6.x86_64
qemu-kvm-tools-0.12.1.2-2.412.el6.x86_64
[root@m2 ~]# rpm -qa |grep glusterfs
glusterfs-3.4.0.34rhs-1.el6.x86_64
glusterfs-api-3.4.0.34rhs-1.el6.x86_64
glusterfs-libs-3.4.0.34rhs-1.el6.x86_64

rhs:
RHS-2.1-20130830.n.0
[root@rhs brick1]# rpm -qa |grep glusterfs
glusterfs-server-3.4.0.34rhs-1.el6rhs.x86_64
samba-glusterfs-3.6.9-160.3.el6rhs.x86_64
glusterfs-3.4.0.34rhs-1.el6rhs.x86_64
glusterfs-fuse-3.4.0.34rhs-1.el6rhs.x86_64
glusterfs-api-3.4.0.34rhs-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.34rhs-1.el6rhs.x86_64
glusterfs-libs-3.4.0.34rhs-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.34rhs-1.el6rhs.x86_64


steps and command line refer to comment#0 and comment#3.

Result:
qemu-kvm run well, no crash no SIGUSR2.

so this bug has been fixed.
Comment 15 errata-xmlrpc 2013-11-21 07:10:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-1553.html
Note You need to log in before you can comment on or make changes to this bug.