Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 880948 - Segfault in write-behind xlator when installing a distro on VM using QEMU-GlusterFS
Summary: Segfault in write-behind xlator when installing a distro on VM using QEMU-Glu...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: write-behind
Version: mainline
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
Assignee: Kaushal
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-11-28 09:09 UTC by Bharata B Rao
Modified: 2013-07-24 17:27 UTC (History)
3 users (show)

Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-07-24 17:27:51 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Bharata B Rao 2012-11-28 09:09:34 UTC
Description of problem:

Installing a VM image on GlusterFS backend using QEMU will result in a segmentation fault in the write-behind xlator code.

Version-Release number of selected component (if applicable):

Latest git

How reproducible:

1. Have a gluster volume with performance.write-behind turned ON
2. qemu-img create -f qcow2 gluster://server/volume/image size
3. qemu-system-x86_64 --enable-kvm -drive file=gluster://server/volume/image,if=virtio,cache=none -cdrom Fedora-DVD.iso
4. Install the distribution on the VM.
 
QEMU segfaults like this:

 Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff297fa700 (LWP 21649)]
0x00007ffff5960368 in __memcpy_ssse3_back () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glib2-2.32.4-2.fc17.x86_64 glibc-2.15-58.fc17.x86_64 gnutls-2.12.20-4.fc17.x86_64 keyutils-libs-1.5.5-2.fc17.x86_64 krb5-libs-1.10.2-6.fc17.x86_64 libaio-0.3.109-5.fc17.x86_64 libcom_err-1.42.3-3.fc17.x86_64 libgcc-4.7.2-2.fc17.x86_64 libgcrypt-1.5.0-3.fc17.x86_64 libgpg-error-1.10-2.fc17.x86_64 libpng-1.5.10-1.fc17.x86_64 libselinux-2.1.10-3.fc17.x86_64 libtasn1-2.12-1.fc17.x86_64 libuuid-2.21.2-2.fc17.x86_64 openssl-1.0.0j-2.fc17.x86_64 p11-kit-0.12-1.fc17.x86_64 pixman-0.24.4-2.fc17.x86_64 zlib-1.2.5-7.fc17.x86_64
(gdb) bt
#0  0x00007ffff5960368 in __memcpy_ssse3_back () from /lib64/libc.so.6
#1  0x00007fffef300dfe in iov_unload (buf=0x7fffee9c4000 "", vector=0x7ffefc000a10, count=1) at ../../../../libglusterfs/src/common-utils.h:344
#2  0x00007fffef303261 in __wb_collapse_small_writes (holder=0x7fff14000a50, req=0x7ffefc001770) at write-behind.c:903
#3  0x00007fffef3034c7 in __wb_preprocess_winds (wb_inode=0x7fffd0000a10) at write-behind.c:979
#4  0x00007fffef303785 in wb_process_queue (wb_inode=0x7fffd0000a10) at write-behind.c:1064
#5  0x00007fffef303c5e in wb_writev (frame=0x5555565ef338, this=0x7fffd4003e20, fd=0x555556b7710c, vector=0x55555c576310, count=1, offset=168361984, flags=
    0, iobref=0x7ffefc0008e0, xdata=0x0) at write-behind.c:1160
#6  0x00007fffef0f794b in ra_writev (frame=0x5555565f0ec4, this=0x7fffd4004980, fd=0x555556b7710c, vector=0x55555c576310, count=1, offset=168361984, flags=
    0, iobref=0x7ffefc0008e0, xdata=0x0) at read-ahead.c:682
#7  0x00007fffeeee0c78 in qr_writev (frame=0x5555565f1174, this=0x7fffd4005320, fd=0x555556b7710c, vector=0x55555c576310, count=1, off=168361984, wr_flags=
    0, iobref=0x7ffefc0008e0, xdata=0x0) at quick-read.c:1525
#8  0x00007fffeecd125f in mdc_writev (frame=0x5555565f31b4, this=0x7fffd4005dd0, fd=0x555556b7710c, vector=0x55555c576310, count=1, offset=168361984, flags=
    0, iobref=0x7ffefc0008e0, xdata=0x0) at md-cache.c:1420
#9  0x00007fffeeabf89b in io_stats_writev (frame=0x5555565f3510, this=0x7fffd4006820, fd=0x555556b7710c, vector=0x55555c576310, count=1, offset=168361984, 
    flags=0, iobref=0x7ffefc0008e0, xdata=0x0) at io-stats.c:2091
#10 0x00007ffff55e94d6 in syncop_writev (subvol=0x7fffd4006820, fd=0x555556b7710c, vector=0x55555c576310, count=1, offset=168361984, iobref=0x7ffefc0008e0, 
    flags=0) at syncop.c:1105
#11 0x00007ffff72a9aeb in glfs_pwritev (glfd=0x555556b92370, iovec=0x55555707d990, iovcnt=2, offset=168361984, flags=0) at glfs-fops.c:576
#12 0x00007ffff72a9619 in glfs_io_async_task (data=0x55555740ed60) at glfs-fops.c:429
#13 0x00007ffff55e28a5 in synctask_wrap (old_task=0x5555573194b0) at syncop.c:129
#14 0x00007ffff5860370 in ?? () from /lib64/libc.so.6
#15 0x0000000000000000 in ?? ()

Comment 2 Amar Tumballi 2012-12-12 11:00:21 UTC
Bharata, next time when you hit this crash (i heard its pretty easy to hit), can you please do below stuff in gdb ?

gdb) fr 2 (ie come into _wb_collapse_small_writes)
gdb) p *req
gdb) p *holder

Thanks

Comment 3 Bharata B Rao 2012-12-12 13:30:27 UTC
(In reply to comment #2)
> Bharata, next time when you hit this crash (i heard its pretty easy to hit),

I am running my volumes with performance.write-behind=off these days to avoid this bug :)

> can you please do below stuff in gdb ?
> 
> gdb) fr 2 (ie come into _wb_collapse_small_writes)
> gdb) p *req
> gdb) p *holder

(gdb) bt
#0  0x00007ffff5960368 in __memcpy_ssse3_back () from /lib64/libc.so.6
#1  0x00007fffeb95ddfe in iov_unload (buf=0x7fffea9f8000 "", vector=0x7fff20000f80, count=1) at ../../../../libglusterfs/src/common-utils.h:344
#2  0x00007fffeb960261 in __wb_collapse_small_writes (holder=0x7fff20000a70, req=0x7fff20001970) at write-behind.c:903
#3  0x00007fffeb9604c7 in __wb_preprocess_winds (wb_inode=0x7fff280013b0) at write-behind.c:979
#4  0x00007fffeb960785 in wb_process_queue (wb_inode=0x7fff280013b0) at write-behind.c:1064
#5  0x00007fffeb960c5e in wb_writev (frame=0x555556612cf4, this=0x7fffd0003e20, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, offset=4572012544, flags=0, iobref=
    0x7fff20001840, xdata=0x0) at write-behind.c:1160
#6  0x00007fffeb75494b in ra_writev (frame=0x555556612da0, this=0x7fffd0004980, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, offset=4572012544, flags=0, iobref=
    0x7fff20001840, xdata=0x0) at read-ahead.c:682
#7  0x00007fffeb53dc78 in qr_writev (frame=0x555556612e4c, this=0x7fffd0005320, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, off=4572012544, wr_flags=0, iobref=
    0x7fff20001840, xdata=0x0) at quick-read.c:1525
#8  0x00007fffeb32e25f in mdc_writev (frame=0x555556613050, this=0x7fffd0005dd0, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, offset=4572012544, flags=0, iobref=
    0x7fff20001840, xdata=0x0) at md-cache.c:1420
#9  0x00007fffeb11c89b in io_stats_writev (frame=0x555556613254, this=0x7fffd0006820, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, offset=4572012544, flags=0, iobref=
    0x7fff20001840, xdata=0x0) at io-stats.c:2091
#10 0x00007ffff55e94d6 in syncop_writev (subvol=0x7fffd0006820, fd=0x555556b99a1c, vector=0x55555922f2c0, count=1, offset=4572012544, iobref=0x7fff20001840, flags=0)
    at syncop.c:1105
#11 0x00007ffff72a9aeb in glfs_pwritev (glfd=0x555556bb4c80, iovec=0x555556f60080, iovcnt=20, offset=4572012544, flags=0) at glfs-fops.c:576
#12 0x00007ffff72a9619 in glfs_io_async_task (data=0x555556c3b1c0) at glfs-fops.c:429
#13 0x00007ffff55e28a5 in synctask_wrap (old_task=0x555556f601d0) at syncop.c:129
#14 0x00007ffff5860370 in ?? () from /lib64/libc.so.6
#15 0x0000000000000000 in ?? ()
(gdb) fr 2
#2  0x00007fffeb960261 in __wb_collapse_small_writes (holder=0x7fff20000a70, req=0x7fff20001970) at write-behind.c:903
903	        iov_unload (ptr, req->stub->args.writev.vector,
(gdb) p *req
$1 = {all = {next = 0x7fff280013d0, prev = 0x7fff20000a70}, todo = {next = 0x7fff280013e0, prev = 0x7fff20000a80}, lie = {next = 0x7fff28001400, prev = 0x7fff20000a90}, 
  winds = {next = 0x7fff200019a0, prev = 0x7fff200019a0}, unwinds = {next = 0x7fff200019b0, prev = 0x7fff200019b0}, stub = 0x5555566bf360, write_size = 8257536, orig_size = 
    8257536, total_size = 0, op_ret = 8257536, op_errno = 0, refcount = 2, wb_inode = 0x7fff280013b0, fop = GF_FOP_WRITE, lk_owner = {len = 0, data = 
    '\000' <repeats 1023 times>}, iobref = 0x0, gen = 1, fd = 0x555556b99a1c, ordering = {size = 8257536, off = 4572012544, append = 0, tempted = -1, lied = 0, fulfilled = 0, 
    go = 0}}
(gdb) p *holder
$2 = {all = {next = 0x7fff20001970, prev = 0x7fff300013b0}, todo = {next = 0x7fff20001980, prev = 0x7fff280013e0}, lie = {next = 0x7fff20001990, prev = 0x7fff28001400}, 
  winds = {next = 0x7fff20000aa0, prev = 0x7fff20000aa0}, unwinds = {next = 0x7fff20000ab0, prev = 0x7fff20000ab0}, stub = 0x5555566bf82c, write_size = 8257536, orig_size = 
    8257536, total_size = 0, op_ret = 8257536, op_errno = 0, refcount = 2, wb_inode = 0x7fff280013b0, fop = GF_FOP_WRITE, lk_owner = {len = 0, data = 
    '\000' <repeats 1023 times>}, iobref = 0x7fff20001e40, gen = 1, fd = 0x555556b99a1c, ordering = {size = 8257536, off = 4563755008, append = 0, tempted = -1, lied = 0, 
    fulfilled = 0, go = 0}}

Comment 4 Vijay Bellur 2012-12-13 22:46:55 UTC
CHANGE: http://review.gluster.org/4307 (write-behind: fixes issues with iobuf length for large writes) merged in master by Anand Avati (avati@redhat.com)


Note You need to log in before you can comment on or make changes to this bug.