Bug 1425296

Summary: qemu_gluster_co_get_block_status gets SIGABRT when doing blockcommit continually
Product: [Community] GlusterFS Reporter: Han Han <hhan>
Component: libgfapiAssignee: Prasanna Kumar Kalever <prasanna.kalever>
Status: CLOSED DUPLICATE QA Contact: Sudhir D <sdharane>
Severity: high Docs Contact:
Priority: unspecified    
Version: mainlineCC: amit, barumuga, berrange, bugs, cfergeau, chhu, crobinso, dwmw2, dyuan, hhan, humble.devassy, itamar, joe, jonathansteffan, juzhang, kkeithle, ndevos, pbonzini, ramkrsna, rjones, vbellur, virt-bugs, virt-maint, virt-maint, xuzhang, yanqzhan
Target Milestone: ---Keywords: Regression, TestBlocker, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1425293 Environment:
Last Closed: 2018-11-20 03:42:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1425293    
Bug Blocks:    

Description Han Han 2017-02-21 06:39:14 UTC 
+++ This bug was initially created as a clone of Bug #1425293 +++

Description of problem:
As subject

Version-Release number of selected component (if applicable):
kernel-3.10.0-514.12.1.el7.x86_64
libvirt-3.0.0-2.el7.x86_64
libvirt-python-3.0.0-1.el7.x86_64
glusterfs-api-3.8.4-15.el7rhgs.x86_64
qemu-kvm-rhev-2.8.0-4.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Setup a glusterfs server.
2. Copy the qcow2 file installed with OS to the glusterfs server.

3. Create a VM whose disk based on the glusterfs server
# DOM=c16572
# cat c16572.xml|awk '/<disk/,/<\/disk/'
    <disk type='network' device='disk'>
      <driver name='qemu' type='qcow2' />
      <source protocol='gluster' name='gluster-vol1/c16572.qcow2'>
        <host name='xx.xx.xx.xx'/>
      </source>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
# virsh create $DOM.xml
Domain c16572 created from c16572.xml

4. Create 2 snapshots, 1 based on glusterfs, 1 based on local
# cat s1.xml 
<domainsnapshot>
<name>s1</name>
<disks>
<disk name='vda' type='network'>
<driver type='qcow2'/>
<source protocol='gluster' name='gluster-vol1/c16572.s1'>
<host name='xx.xx.xx.xx'/>
</source>
</disk>
</disks>
</domainsnapshot>

# virsh snapshot-create $DOM s1.xml --disk-only
Domain snapshot s1 created from 's1.xml'
# virsh snapshot-create-as $DOM s2 --disk-only --diskspec vda,file=/var/lib/libvirt/images/$DOM.s2
Domain snapshot s2 created

5. Do blockcommit&blockjob --pivot continually
# virsh blockcommit $DOM vda --active --shallow --wait --verbose
Block commit: [100 %]
Now in synchronized phase
# virsh dumpxml $DOM|awk '/<disk/,/<\/disk/'
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/c16572.s2'/>
      <backingStore type='network' index='1'>
        <format type='qcow2'/>
        <source protocol='gluster' name='gluster-vol1/c16572.s1'>
          <host name='xx.xx.xx.xx'/>
        </source>
        <backingStore type='network' index='2'>
          <format type='qcow2'/>
          <source protocol='gluster' name='gluster-vol1/c16572.qcow2'>
            <host name='xx.xx.xx.xx'/>
          </source>
          <backingStore/>
        </backingStore>
      </backingStore>
      <mirror type='network' job='active-commit' ready='yes'>
        <format type='qcow2'/>
        <source protocol='gluster' name='gluster-vol1/c16572.s1'>
          <host name='xx.xx.xx.xx'/>
        </source>
      </mirror>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>

# virsh blockjob $DOM vda --pivot

# virsh dumpxml $DOM|awk '/<disk/,/<\/disk/'
 <disk type='network' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source protocol='gluster' name='gluster-vol1/c16572.s1'>
        <host name='xx.xx.xx.xx'/>
      </source>
      <backingStore type='network' index='1'>
        <format type='qcow2'/>
        <source protocol='gluster' name='gluster-vol1/c16572.qcow2'>
          <host name='xx.xx.xx.xx'/>
        </source>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>

# virsh blockcommit $DOM vda --active --shallow --wait --verbose
error: internal error: child reported: Kernel does not provide mount namespace: No such file or directory
# virsh dumpxml $DOM|awk '/<disk/,/<\/disk/'
error: failed to get domain 'c16572'
error: Domain not found: no domain with matching name 'c16572'
# virsh blockjob $DOM vda --pivot
error: failed to get domain 'c16572'
error: Domain not found: no domain with matching name 'c16572'
# virsh dumpxml $DOM|awk '/<disk/,/<\/disk/'
error: failed to get domain 'c16572'
error: Domain not found: no domain with matching name 'c16572'

6.
There is coredump:
# id 98246f231004278367ef68284bed2d3823a699cb
reason:         qemu-kvm killed by SIGABRT
time:           Tue 21 Feb 2017 11:24:15 AM CST
cmdline:        /usr/libexec/qemu-kvm -name guest=16572,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-8-16572/master-key.aes -machine pc-i440fx-rhel7.2.0,accel=kvm
,usb=off,vmport=off,dump-guest-core=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid ad912fb3-9e5e-4426-9e13-a9ab5f72e207 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/
var/lib/libvirt/qemu/domain-8-16572/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PI
IX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -devi
ce ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -driv
e file=gluster://xx.xx.xx.xx/gluster-vol1/16572.qcow2,file.debug=4,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0
,bootindex=1 -netdev tap,fd=29,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:4e:f5:57,bus=pci.0,addr=0x9 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev 
spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channel/
target/domain-8-16572/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,por
t=1 -spice port=5901,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -
device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 
-chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
package:        qemu-kvm-rhev-2.8.0-4.el7
uid:            107 (qemu)
count:          9
Directory:      /var/spool/abrt/ccpp-2017-02-20-22:24:15-9498
Run 'abrt-cli report /var/spool/abrt/ccpp-2017-02-20-22:24:15-9498' for creating a case in Red Hat Customer Portal

The backtrace of coredump:
(gdb) bt
#0  0x00007ffb4483d1d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffb4483e8c8 in __GI_abort () at abort.c:90
#2  0x00007ffb44836146 in __assert_fail_base (fmt=0x7ffb44987428 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7ffb5da443f4 "offs >= start", file=file@entry=0x7ffb5da49608 "block/gluster.c", line=line@entry=1285, function=function@entry=0x7ffb5da49d40 <__PRETTY_FUNCTION__.24238> "find_allocation") at assert.c:92
#3  0x00007ffb448361f2 in __GI___assert_fail (assertion=assertion@entry=0x7ffb5da443f4 "offs >= start", file=file@entry=0x7ffb5da49608 "block/gluster.c", line=line@entry=1285, function=function@entry=0x7ffb5da49d40 <__PRETTY_FUNCTION__.24238> "find_allocation") at assert.c:101
#4  0x00007ffb5d930ca2 in qemu_gluster_co_get_block_status (bs=0x7ffb61ed6800, hole=<synthetic pointer>, data=<synthetic pointer>, start=2555904) at block/gluster.c:1285
#5  0x00007ffb5d930ca2 in qemu_gluster_co_get_block_status (bs=0x7ffb61ed6800, sector_num=<optimized out>, nb_sectors=4224, pnum=0x7ffae34d8cfc, file=0x7ffae34d8d00) at block/gluster.c:1378
#6  0x00007ffb5d918236 in bdrv_co_get_block_status (bs=0x7ffb61ed6800, sector_num=sector_num@entry=4992, nb_sectors=4224, pnum=pnum@entry=0x7ffae34d8cfc, file=file@entry=0x7ffae34d8d00) at block/io.c:1749
#7  0x00007ffb5d91839a in bdrv_co_get_block_status (bs=bs@entry=0x7ffb61534800, sector_num=sector_num@entry=1054080, nb_sectors=<optimized out>, 
    nb_sectors@entry=4194303, pnum=pnum@entry=0x7ffae34d8e34, file=file@entry=0x7ffae34d8e00) at block/io.c:1783
#8  0x00007ffb5d91843b in bdrv_get_block_status_above_co_entry (file=0x7ffae34d8e00, pnum=0x7ffae34d8e34, nb_sectors=4194303, sector_num=1054080, base=0x7ffb5fd0e000, bs=<optimized out>) at block/io.c:1819
#9  0x00007ffb5d91843b in bdrv_get_block_status_above_co_entry (opaque=opaque@entry=0x7ffae34d8d90) at block/io.c:1835
#10 0x00007ffb5d918138 in bdrv_get_block_status_above (bs=0x7ffb61534800, base=<optimized out>, sector_num=sector_num@entry=1054080, nb_sectors=nb_sectors@entry=4194303, pnum=<optimized out>, file=file@entry=0x7ffae34d8e00) at block/io.c:1867
#11 0x00007ffb5d918516 in bdrv_is_allocated (file=0x7ffae34d8e00, pnum=0x7ffae34d8e34, nb_sectors=4194303, sector_num=1054080, bs=0x7ffb61534800) at block/io.c:1882
#12 0x00007ffb5d918516 in bdrv_is_allocated (bs=bs@entry=0x7ffb61534800, sector_num=sector_num@entry=1054080, nb_sectors=nb_sectors@entry=4194303, pnum=pnum@entry=0x7ffae34d8e34) at block/io.c:1890
#13 0x00007ffb5d9185b1 in bdrv_is_allocated_above (top=top@entry=0x7ffb61534800, base=base@entry=0x7ffb5fd0e000, sector_num=sector_num@entry=1054080, nb_sectors=nb_sectors@entry=4194303, pnum=pnum@entry=0x7ffae34d8f48) at block/io.c:1921
#14 0x00007ffb5d915ac6 in mirror_run (s=0x7ffb5fc2bee0) at block/mirror.c:604
#15 0x00007ffb5d915ac6 in mirror_run (opaque=0x7ffb5fc2bee0) at block/mirror.c:695
#16 0x00007ffb5d98a55a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at util/coroutine-ucontext.c:79
#17 0x00007ffb4484ecf0 in __start_context () at /usr/lib64/libc-2.17.so
#18 0x00007ffcf69a1900 in  ()
#19 0x0000000000000000 in  ()



Actual results:
qemu gest SIGABRT 

Expected results:
NO SIGABRT

Additional info:
Log and reproducing scripts are in the attachment.
Qemu VM log:
# cat /var/log/libvirt/qemu/c16572.log                                                                                                                                                                  
2017-02-21 06:00:13.610+0000: starting up libvirt version: 3.0.0, package: 2.el7 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2017-02-09-04:47:33, x86-037.build.eng.bos.redhat.com), qemu version: 2.8.0(qemu-kvm-rhev-2.8.0-4.el7), hostname: lab.work.me
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=c16572,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-32-c16572/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Penryn -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 9abaaea3-e69b-4a9d-bce6-841f03763474 -no-user-config -nodefaults -charde
v socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-32-c16572/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,mast
erbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=gluster://xx.xx.xx.xx/gluster-vol1/c16572.qcow2,file.debug=4,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=31,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac
=52:54:00:ad:71:b5,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5901,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -
device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
2017-02-21 06:00:13.696+0000: 25103: debug : virFileClose:109 : Closed fd 32
2017-02-21 06:00:13.696+0000: 25103: debug : virFileClose:109 : Closed fd 37
2017-02-21 06:00:13.726+0000: 25103: debug : virFileClose:109 : Closed fd 3
2017-02-21 06:00:13.728+0000: 25104: debug : virExec:697 : Run hook 0x7f2e5ec78a80 0x7f2e78561840
2017-02-21 06:00:13.728+0000: 25104: debug : qemuProcessHook:2646 : Obtaining domain lock
2017-02-21 06:00:13.728+0000: 25104: debug : virSecuritySELinuxSetSocketLabel:2574 : Setting VM c16572 socket context system_u:system_r:svirt_t:s0:c232,c920
2017-02-21 06:00:13.729+0000: 25104: debug : virDomainLockProcessStart:179 : plugin=0x7f2e141276e0 dom=0x7f2e689d4c60 paused=1 fd=0x7f2e78561370
2017-02-21 06:00:13.729+0000: 25104: debug : virDomainLockManagerNew:134 : plugin=0x7f2e141276e0 dom=0x7f2e689d4c60 withResources=1
2017-02-21 06:00:13.729+0000: 25104: debug : virLockManagerPluginGetDriver:281 : plugin=0x7f2e141276e0
2017-02-21 06:00:13.729+0000: 25104: debug : virLockManagerNew:305 : driver=0x7f2e1edc4000 type=0 nparams=5 params=0x7f2e78561220 flags=1
2017-02-21 06:00:13.730+0000: 25104: debug : virLockManagerLogParams:98 :   key=uuid type=uuid value=9abaaea3-e69b-4a9d-bce6-841f03763474
2017-02-21 06:00:13.730+0000: 25104: debug : virLockManagerLogParams:91 :   key=name type=string value=c16572
2017-02-21 06:00:13.730+0000: 25104: debug : virLockManagerLogParams:79 :   key=id type=uint value=32
2017-02-21 06:00:13.730+0000: 25104: debug : virLockManagerLogParams:79 :   key=pid type=uint value=25104
2017-02-21 06:00:13.730+0000: 25104: debug : virLockManagerLogParams:94 :   key=uri type=cstring value=qemu:///system
2017-02-21 06:00:13.730+0000: 25104: debug : virDomainLockManagerNew:146 : Adding leases
2017-02-21 06:00:13.730+0000: 25104: debug : virDomainLockManagerNew:151 : Adding disks
2017-02-21 06:00:13.730+0000: 25104: debug : virLockManagerAcquire:350 : lock=0x7f2e68a89990 state='<null>' flags=3 action=0 fd=0x7f2e78561370
2017-02-21 06:00:13.730+0000: 25104: debug : virLockManagerSanlockAcquire:935 : Register sanlock 3
2017-02-21 06:00:13.731+0000: 25104: debug : virLockManagerSanlockAcquire:1029 : Acquire completed fd=3
2017-02-21 06:00:13.731+0000: 25104: debug : virLockManagerFree:387 : lock=0x7f2e68a89990
2017-02-21 06:00:13.731+0000: 25104: info : virObjectRef:296 : OBJECT_REF: obj=0x7f2e141b83d0
2017-02-21 06:00:13.731+0000: 25104: info : virObjectRef:296 : OBJECT_REF: obj=0x7f2e141b83d0
2017-02-21 06:00:13.731+0000: 25104: debug : virFileGetMountSubtreeImpl:1921 : prefix=/dev
2017-02-21 06:00:13.732+0000: 25104: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f2e141b83d0
2017-02-21 06:00:13.732+0000: 25104: debug : qemuDomainSetupDev:7194 : Setting up /dev/ for domain c16572
2017-02-21 06:00:13.732+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.732+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu mode=0777
2017-02-21 06:00:13.732+0000: 25104: debug : virFileSetupDev:3595 : Mount devfs on /var/run/libvirt/qemu/c16572.dev type=tmpfs flags=2, opts=mode=755,size=65536
2017-02-21 06:00:13.732+0000: 25104: info : virObjectRef:296 : OBJECT_REF: obj=0x7f2e141b83d0
2017-02-21 06:00:13.732+0000: 25104: debug : virFileMakeParentPath:2982 : path=/var/run/libvirt/qemu/c16572.dev/null
2017-02-21 06:00:13.732+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.733+0000: 25104: debug : virFileMakeParentPath:2982 : path=/var/run/libvirt/qemu/c16572.dev/full
2017-02-21 06:00:13.733+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.734+0000: 25104: debug : virFileMakeParentPath:2982 : path=/var/run/libvirt/qemu/c16572.dev/zero
2017-02-21 06:00:13.734+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.734+0000: 25104: debug : virFileMakeParentPath:2982 : path=/var/run/libvirt/qemu/c16572.dev/random
2017-02-21 06:00:13.734+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.734+0000: 25104: debug : virFileMakeParentPath:2982 : path=/var/run/libvirt/qemu/c16572.dev/urandom
2017-02-21 06:00:13.734+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.734+0000: 25104: debug : virFileMakeParentPath:2982 : path=/var/run/libvirt/qemu/c16572.dev/ptmx
2017-02-21 06:00:13.734+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.734+0000: 25104: debug : virFileMakeParentPath:2982 : path=/var/run/libvirt/qemu/c16572.dev/kvm
2017-02-21 06:00:13.734+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.735+0000: 25104: debug : virFileMakeParentPath:2982 : path=/var/run/libvirt/qemu/c16572.dev/rtc
2017-02-21 06:00:13.735+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.735+0000: 25104: debug : virFileMakeParentPath:2982 : path=/var/run/libvirt/qemu/c16572.dev/rtc0
2017-02-21 06:00:13.735+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.735+0000: 25104: debug : virFileMakeParentPath:2982 : path=/var/run/libvirt/qemu/c16572.dev/hpet
2017-02-21 06:00:13.735+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.735+0000: 25104: debug : virFileMakeParentPath:2982 : path=/var/run/libvirt/qemu/c16572.dev/vfio/vfio
2017-02-21 06:00:13.735+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev/vfio mode=0777
2017-02-21 06:00:13.735+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.dev mode=0777
2017-02-21 06:00:13.736+0000: 25104: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f2e141b83d0
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.hugepages mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.mqueue mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.pts mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu/c16572.shm mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/var/run/libvirt/qemu mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : qemuDomainSetupAllDisks:7257 : Setting up disks
2017-02-21 06:00:13.736+0000: 25104: debug : qemuDomainSetupAllDisks:7266 : Setup all disks
2017-02-21 06:00:13.736+0000: 25104: debug : qemuDomainSetupAllHostdevs:7305 : Setting up hostdevs
2017-02-21 06:00:13.736+0000: 25104: debug : qemuDomainSetupAllHostdevs:7312 : Setup all hostdevs
2017-02-21 06:00:13.736+0000: 25104: debug : qemuDomainSetupAllChardevs:7336 : Setting up chardevs
2017-02-21 06:00:13.736+0000: 25104: debug : qemuDomainSetupAllChardevs:7344 : Setup all chardevs
2017-02-21 06:00:13.736+0000: 25104: debug : qemuDomainSetupAllInputs:7412 : Setting up inputs
2017-02-21 06:00:13.736+0000: 25104: debug : qemuDomainSetupAllInputs:7419 : Setup all inputs
2017-02-21 06:00:13.736+0000: 25104: debug : qemuDomainSetupAllRNGs:7451 : Setting up RNGs
2017-02-21 06:00:13.736+0000: 25104: debug : qemuDomainSetupAllRNGs:7459 : Setup all RNGs
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/dev/hugepages mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/dev mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/dev/mqueue mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/dev mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/dev/pts mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/dev mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/dev/shm mode=0777
2017-02-21 06:00:13.736+0000: 25104: debug : virFileMakePathHelper:2910 : path=/dev mode=0777
2017-02-21 06:00:13.736+0000: 25104: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f2e141b83d0
2017-02-21 06:00:13.736+0000: 25104: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f2e141b83d0
2017-02-21 06:00:13.736+0000: 25104: debug : qemuProcessHook:2690 : Hook complete ret=0
2017-02-21 06:00:13.737+0000: 25104: debug : virExec:699 : Done hook 0
2017-02-21 06:00:13.737+0000: 25104: debug : virExec:706 : Setting child security label to system_u:system_r:svirt_t:s0:c232,c920
2017-02-21 06:00:13.737+0000: 25104: debug : virExec:736 : Setting child uid:gid to 107:107 with caps 0
2017-02-21 06:00:13.737+0000: 25104: debug : virCommandHandshakeChild:435 : Notifying parent for handshake start on 34
2017-02-21 06:00:13.737+0000: 25104: debug : virCommandHandshakeChild:443 : Waiting on parent for handshake complete on 35
2017-02-21 06:00:14.905+0000: 25104: debug : virFileClose:109 : Closed fd 34
2017-02-21 06:00:14.905+0000: 25104: debug : virFileClose:109 : Closed fd 35
2017-02-21 06:00:14.905+0000: 25104: debug : virCommandHandshakeChild:463 : Handshake with parent is done
char device redirected to /dev/pts/41 (label charserial0)
[2017-02-21 06:00:15.055594] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 6c61622e-776f-726b-2e6d-652d32353130 (0) coming up
[2017-02-21 06:00:15.055654] I [MSGID: 114020] [client.c:2356:notify] 0-gluster-vol1-client-0: parent translators are ready, attempting connect on transport
[2017-02-21 06:00:15.059810] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-gluster-vol1-client-0: changing port to 49152 (from 0)
[2017-02-21 06:00:15.063487] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 0-gluster-vol1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-02-21 06:00:15.064025] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 0-gluster-vol1-client-0: Connected to gluster-vol1-client-0, attached to remote volume '/br1'.
[2017-02-21 06:00:15.064041] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 0-gluster-vol1-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-02-21 06:00:15.086593] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gluster-vol1-client-0: Server lk version = 1
[2017-02-21 06:00:15.087550] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-gluster-vol1: switched to graph 6c61622e-776f-726b-2e6d-652d32353130 (0)
[2017-02-21 06:00:15.089021] W [MSGID: 114031] [client-rpc-fops.c:2211:client3_3_seek_cbk] 0-gluster-vol1-client-0: remote operation failed [No such device or address]
Formatting 'gluster://xx.xx.xx.xx/gluster-vol1/c16572.s1', fmt=qcow2 size=10737418240 backing_file=gluster://xx.xx.xx.xx/gluster-vol1/c16572.qcow2 backing_fmt=qcow2 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
Formatting '/var/lib/libvirt/images/c16572.s2', fmt=qcow2 size=10737418240 backing_file=gluster://xx.xx.xx.xx/gluster-vol1/c16572.s1 backing_fmt=qcow2 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
qemu-kvm: block/gluster.c:1285: find_allocation: Assertion `offs >= start' failed.
2017-02-21 06:00:30.320+0000: shutting down, reason=crashed

--- Additional comment from Han Han on 2017-02-21 01:16:25 EST ---

I did't encounter the bug when testing with libvirt-2.5.0-1.el7 and qemu-kvm-rhev-2.6.0-29.el7 one month ago. Moreover, it blocks a case of libvirt. Marked as regression and testblocker.
Comment 1 Han Han 2017-02-21 06:40:27 UTC 
Reproduced on qemu-kvm-2.8.0-1.fc26.x86_64 libvirt-3.1.0-1.fc26.x86_64
Comment 2 Fedora End Of Life 2017-02-28 12:23:54 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.
Comment 3 Cole Robinson 2017-03-19 20:22:32 UTC 
There's a qemu 2.9.0 build for f26/rawhide now, can you still reproduce?
Comment 4 Han Han 2017-03-20 02:11:13 UTC 
This bug could be reproduced on qemu-system-x86-core-2.9.0-0.1.rc0.fc26, glusterfs-3.10.0-1.fc26.x86_64

According to BZ1425293, it should be the bug of glusterfs.
Comment 5 Vijay Bellur 2018-11-20 03:42:41 UTC 

*** This bug has been marked as a duplicate of bug 1398076 ***