Bug 880948
| Summary: | Segfault in write-behind xlator when installing a distro on VM using QEMU-GlusterFS | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Bharata B Rao <bharata.rao> |
| Component: | write-behind | Assignee: | Kaushal <kaushal> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | mainline | CC: | aavati, amarts, gluster-bugs |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | glusterfs-3.4.0 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-07-24 17:27:51 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Bharata, next time when you hit this crash (i heard its pretty easy to hit), can you please do below stuff in gdb ? gdb) fr 2 (ie come into _wb_collapse_small_writes) gdb) p *req gdb) p *holder Thanks (In reply to comment #2) > Bharata, next time when you hit this crash (i heard its pretty easy to hit), I am running my volumes with performance.write-behind=off these days to avoid this bug :) > can you please do below stuff in gdb ? > > gdb) fr 2 (ie come into _wb_collapse_small_writes) > gdb) p *req > gdb) p *holder (gdb) bt #0 0x00007ffff5960368 in __memcpy_ssse3_back () from /lib64/libc.so.6 #1 0x00007fffeb95ddfe in iov_unload (buf=0x7fffea9f8000 "", vector=0x7fff20000f80, count=1) at ../../../../libglusterfs/src/common-utils.h:344 #2 0x00007fffeb960261 in __wb_collapse_small_writes (holder=0x7fff20000a70, req=0x7fff20001970) at write-behind.c:903 #3 0x00007fffeb9604c7 in __wb_preprocess_winds (wb_inode=0x7fff280013b0) at write-behind.c:979 #4 0x00007fffeb960785 in wb_process_queue (wb_inode=0x7fff280013b0) at write-behind.c:1064 #5 0x00007fffeb960c5e in wb_writev (frame=0x555556612cf4, this=0x7fffd0003e20, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, offset=4572012544, flags=0, iobref= 0x7fff20001840, xdata=0x0) at write-behind.c:1160 #6 0x00007fffeb75494b in ra_writev (frame=0x555556612da0, this=0x7fffd0004980, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, offset=4572012544, flags=0, iobref= 0x7fff20001840, xdata=0x0) at read-ahead.c:682 #7 0x00007fffeb53dc78 in qr_writev (frame=0x555556612e4c, this=0x7fffd0005320, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, off=4572012544, wr_flags=0, iobref= 0x7fff20001840, xdata=0x0) at quick-read.c:1525 #8 0x00007fffeb32e25f in mdc_writev (frame=0x555556613050, this=0x7fffd0005dd0, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, offset=4572012544, flags=0, iobref= 0x7fff20001840, xdata=0x0) at md-cache.c:1420 #9 0x00007fffeb11c89b in io_stats_writev (frame=0x555556613254, this=0x7fffd0006820, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, offset=4572012544, flags=0, iobref= 0x7fff20001840, xdata=0x0) at io-stats.c:2091 #10 0x00007ffff55e94d6 in syncop_writev (subvol=0x7fffd0006820, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, offset=4572012544, iobref=0x7fff20001840, flags=0) at syncop.c:1105 #11 0x00007ffff72a9aeb in glfs_pwritev (glfd=0x555556bb4c80, iovec=0x555556f60080, iovcnt=20, offset=4572012544, flags=0) at glfs-fops.c:576 #12 0x00007ffff72a9619 in glfs_io_async_task (data=0x555556c3b1c0) at glfs-fops.c:429 #13 0x00007ffff55e28a5 in synctask_wrap (old_task=0x555556f601d0) at syncop.c:129 #14 0x00007ffff5860370 in ?? () from /lib64/libc.so.6 #15 0x0000000000000000 in ?? () (gdb) fr 2 #2 0x00007fffeb960261 in __wb_collapse_small_writes (holder=0x7fff20000a70, req=0x7fff20001970) at write-behind.c:903 903 iov_unload (ptr, req->stub->args.writev.vector, (gdb) p *req $1 = {all = {next = 0x7fff280013d0, prev = 0x7fff20000a70}, todo = {next = 0x7fff280013e0, prev = 0x7fff20000a80}, lie = {next = 0x7fff28001400, prev = 0x7fff20000a90}, winds = {next = 0x7fff200019a0, prev = 0x7fff200019a0}, unwinds = {next = 0x7fff200019b0, prev = 0x7fff200019b0}, stub = 0x5555566bf360, write_size = 8257536, orig_size = 8257536, total_size = 0, op_ret = 8257536, op_errno = 0, refcount = 2, wb_inode = 0x7fff280013b0, fop = GF_FOP_WRITE, lk_owner = {len = 0, data = '\000' <repeats 1023 times>}, iobref = 0x0, gen = 1, fd = 0x555556b99a1c, ordering = {size = 8257536, off = 4572012544, append = 0, tempted = -1, lied = 0, fulfilled = 0, go = 0}} (gdb) p *holder $2 = {all = {next = 0x7fff20001970, prev = 0x7fff300013b0}, todo = {next = 0x7fff20001980, prev = 0x7fff280013e0}, lie = {next = 0x7fff20001990, prev = 0x7fff28001400}, winds = {next = 0x7fff20000aa0, prev = 0x7fff20000aa0}, unwinds = {next = 0x7fff20000ab0, prev = 0x7fff20000ab0}, stub = 0x5555566bf82c, write_size = 8257536, orig_size = 8257536, total_size = 0, op_ret = 8257536, op_errno = 0, refcount = 2, wb_inode = 0x7fff280013b0, fop = GF_FOP_WRITE, lk_owner = {len = 0, data = '\000' <repeats 1023 times>}, iobref = 0x7fff20001e40, gen = 1, fd = 0x555556b99a1c, ordering = {size = 8257536, off = 4563755008, append = 0, tempted = -1, lied = 0, fulfilled = 0, go = 0}} CHANGE: http://review.gluster.org/4307 (write-behind: fixes issues with iobuf length for large writes) merged in master by Anand Avati (avati) |
Description of problem: Installing a VM image on GlusterFS backend using QEMU will result in a segmentation fault in the write-behind xlator code. Version-Release number of selected component (if applicable): Latest git How reproducible: 1. Have a gluster volume with performance.write-behind turned ON 2. qemu-img create -f qcow2 gluster://server/volume/image size 3. qemu-system-x86_64 --enable-kvm -drive file=gluster://server/volume/image,if=virtio,cache=none -cdrom Fedora-DVD.iso 4. Install the distribution on the VM. QEMU segfaults like this: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff297fa700 (LWP 21649)] 0x00007ffff5960368 in __memcpy_ssse3_back () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glib2-2.32.4-2.fc17.x86_64 glibc-2.15-58.fc17.x86_64 gnutls-2.12.20-4.fc17.x86_64 keyutils-libs-1.5.5-2.fc17.x86_64 krb5-libs-1.10.2-6.fc17.x86_64 libaio-0.3.109-5.fc17.x86_64 libcom_err-1.42.3-3.fc17.x86_64 libgcc-4.7.2-2.fc17.x86_64 libgcrypt-1.5.0-3.fc17.x86_64 libgpg-error-1.10-2.fc17.x86_64 libpng-1.5.10-1.fc17.x86_64 libselinux-2.1.10-3.fc17.x86_64 libtasn1-2.12-1.fc17.x86_64 libuuid-2.21.2-2.fc17.x86_64 openssl-1.0.0j-2.fc17.x86_64 p11-kit-0.12-1.fc17.x86_64 pixman-0.24.4-2.fc17.x86_64 zlib-1.2.5-7.fc17.x86_64 (gdb) bt #0 0x00007ffff5960368 in __memcpy_ssse3_back () from /lib64/libc.so.6 #1 0x00007fffef300dfe in iov_unload (buf=0x7fffee9c4000 "", vector=0x7ffefc000a10, count=1) at ../../../../libglusterfs/src/common-utils.h:344 #2 0x00007fffef303261 in __wb_collapse_small_writes (holder=0x7fff14000a50, req=0x7ffefc001770) at write-behind.c:903 #3 0x00007fffef3034c7 in __wb_preprocess_winds (wb_inode=0x7fffd0000a10) at write-behind.c:979 #4 0x00007fffef303785 in wb_process_queue (wb_inode=0x7fffd0000a10) at write-behind.c:1064 #5 0x00007fffef303c5e in wb_writev (frame=0x5555565ef338, this=0x7fffd4003e20, fd=0x555556b7710c, vector=0x55555c576310, count=1, offset=168361984, flags= 0, iobref=0x7ffefc0008e0, xdata=0x0) at write-behind.c:1160 #6 0x00007fffef0f794b in ra_writev (frame=0x5555565f0ec4, this=0x7fffd4004980, fd=0x555556b7710c, vector=0x55555c576310, count=1, offset=168361984, flags= 0, iobref=0x7ffefc0008e0, xdata=0x0) at read-ahead.c:682 #7 0x00007fffeeee0c78 in qr_writev (frame=0x5555565f1174, this=0x7fffd4005320, fd=0x555556b7710c, vector=0x55555c576310, count=1, off=168361984, wr_flags= 0, iobref=0x7ffefc0008e0, xdata=0x0) at quick-read.c:1525 #8 0x00007fffeecd125f in mdc_writev (frame=0x5555565f31b4, this=0x7fffd4005dd0, fd=0x555556b7710c, vector=0x55555c576310, count=1, offset=168361984, flags= 0, iobref=0x7ffefc0008e0, xdata=0x0) at md-cache.c:1420 #9 0x00007fffeeabf89b in io_stats_writev (frame=0x5555565f3510, this=0x7fffd4006820, fd=0x555556b7710c, vector=0x55555c576310, count=1, offset=168361984, flags=0, iobref=0x7ffefc0008e0, xdata=0x0) at io-stats.c:2091 #10 0x00007ffff55e94d6 in syncop_writev (subvol=0x7fffd4006820, fd=0x555556b7710c, vector=0x55555c576310, count=1, offset=168361984, iobref=0x7ffefc0008e0, flags=0) at syncop.c:1105 #11 0x00007ffff72a9aeb in glfs_pwritev (glfd=0x555556b92370, iovec=0x55555707d990, iovcnt=2, offset=168361984, flags=0) at glfs-fops.c:576 #12 0x00007ffff72a9619 in glfs_io_async_task (data=0x55555740ed60) at glfs-fops.c:429 #13 0x00007ffff55e28a5 in synctask_wrap (old_task=0x5555573194b0) at syncop.c:129 #14 0x00007ffff5860370 in ?? () from /lib64/libc.so.6 #15 0x0000000000000000 in ?? ()