Created attachment 1385817 [details] the ansible bundle Issue : Git clone --bare --mirror of git bundle fails when cloning on gluster storage Issue description ------ The customer is encountering the following issue: $ git clone --bare --mirror ../ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects After a few retries it eventually succeeds: $ git clone --bare --mirror ../ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Is reproducible ? Yes, below are my steps: Create a replica 3 volume. Copy the ansible.bundle to /tmp. Run a while loop: [root@v3317a mnt]# mount | grep glusterfs v3317a:/rep3-vol on /mnt type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) [root@v3317a mnt]# while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. ^C =-=-=-=-= Customer's Environment details : GlusterFS version (from all nodes) Kernel version used (if required) I was able to reproduced this on my 3.3.1 async. I could not reproduce this on the local FS. I have attached the ansible bundle file that is causing grief.
Looks like the issue is similar to the other DB consistency issues we saw. Can it be tried with disable open-behind translator (and if that doesn't work, try with disabling other perf translators. Also, if its reproducible, can we add this as a test case in QE testing?
Hello Amar, Reproducer steps: 1) create and start a replica 3 volume 2) copy ansible.bundle to /tmp 3) mount the volume onto /mnt 4) run while loop # while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done I am trying out your suggestions.
Turn off open-behind did not help: [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle tempCloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects [root@v3317a mnt]# gluster volume set tprep3-vol open-behind off volume set: success [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects
Works with write-behind: [root@v3317a mnt]# gluster v info tprep3-vol Volume Name: tprep3-vol Type: Replicate Volume ID: d70f3972-e48e-4fb9-b8fa-39a64190bd0c Status: Started Snapshot Count: 2 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: v3317a:/rhs/brick2/threp3brick Brick2: v3317b:/rhs/brick2/threp3brick Brick3: v3317c:/rhs/brick2/threp3brick Options Reconfigured: nfs.disable: on [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects [root@v3317a mnt]# gluster volume set tprep3-vol performance.write-behind on volume set: success [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects [root@v3317a mnt]# gluster volume set tprep3-vol performance.write-behind off volume set: success [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle temp fatal: destination path 'temp' already exists and is not an empty directory. [root@v3317a mnt]# rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. [root@v3317a mnt]# rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. =-=-=-=-= With the while loop =-=-=-=-= [root@v3317a mnt]# while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. ^C
while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Validated this bug as per the reproducer steps mentioned above. The issue is resolved. Marking this verified against the below version. Version: --------------- glusterfs-server-3.12.2-8.el7rhgs.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607