Bug 1538366 - [GSS] Git clone --bare --mirror of git bundle fails when cloning on gluster storage
Summary: [GSS] Git clone --bare --mirror of git bundle fails when cloning on gluster s...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: write-behind
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
high
low
Target Milestone: ---
: RHGS 3.4.0
Assignee: Csaba Henk
QA Contact: Vivek Das
URL:
Whiteboard:
Depends On:
Blocks: 1503138
TreeView+ depends on / blocked
 
Reported: 2018-01-24 22:59 UTC by Oonkwee Lim_
Modified: 2018-09-11 06:34 UTC (History)
8 users (show)

Fixed In Version: glusterfs-3.12.2-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-04 06:42:04 UTC


Attachments (Terms of Use)
the ansible bundle (1.83 KB, text/plain)
2018-01-24 22:59 UTC, Oonkwee Lim_
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2607 None None None 2018-09-04 06:43:39 UTC

Description Oonkwee Lim_ 2018-01-24 22:59:47 UTC
Created attachment 1385817 [details]
the ansible bundle

Issue : Git clone --bare --mirror of git bundle fails when cloning on gluster storage

Issue description ------

The customer is encountering the following issue:

$ git clone --bare --mirror ../ansible.bundle temp
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
fatal: premature end of pack file, 28 bytes missing
error: index-pack died
fatal: bad object bf4086161ade363a3a851805690f5babb24e9914
fatal: remote did not send all necessary objects

After a few retries it eventually succeeds:

$ git clone --bare --mirror ../ansible.bundle temp
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.


Is reproducible ?  Yes, below are my steps:

Create a replica 3 volume.

Copy the ansible.bundle to /tmp.

Run a while loop:

[root@v3317a mnt]# mount | grep glusterfs
v3317a:/rep3-vol on /mnt type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@v3317a mnt]# while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
fatal: premature end of pack file, 28 bytes missing
error: index-pack died
fatal: bad object bf4086161ade363a3a851805690f5babb24e9914
fatal: remote did not send all necessary objects
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
fatal: premature end of pack file, 28 bytes missing
error: index-pack died
fatal: bad object bf4086161ade363a3a851805690f5babb24e9914
fatal: remote did not send all necessary objects
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
fatal: premature end of pack file, 28 bytes missing
error: index-pack died
fatal: bad object bf4086161ade363a3a851805690f5babb24e9914
fatal: remote did not send all necessary objects
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
^C
=-=-=-=-=
Customer's Environment details :
GlusterFS version (from all nodes)
Kernel version used (if required)

I was able to reproduced  this on my 3.3.1 async.
I could not reproduce this on the local FS.

I have attached the ansible bundle file that is causing grief.

Comment 2 Amar Tumballi 2018-01-30 12:21:40 UTC
Looks like the issue is similar to the other DB consistency issues we saw. 

Can it be tried with disable open-behind translator (and if that doesn't work, try with disabling other perf translators.

Also, if its reproducible, can we add this as a test case in QE testing?

Comment 3 Oonkwee Lim_ 2018-01-31 03:10:51 UTC
Hello Amar,

Reproducer steps:

1) create and start a replica 3 volume
2) copy ansible.bundle to /tmp
3) mount the volume onto /mnt
4) run while loop
# while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done

I am trying out your suggestions.

Comment 4 Oonkwee Lim_ 2018-01-31 03:19:29 UTC
Turn off open-behind did not help:

[root@v3317a mnt]#  git clone --bare --mirror /tmp/ansible.bundle tempCloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
fatal: premature end of pack file, 28 bytes missing
error: index-pack died
fatal: bad object bf4086161ade363a3a851805690f5babb24e9914
fatal: remote did not send all necessary objects
[root@v3317a mnt]# gluster volume set tprep3-vol open-behind off
volume set: success
[root@v3317a mnt]#  git clone --bare --mirror /tmp/ansible.bundle temp
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
fatal: premature end of pack file, 28 bytes missing
error: index-pack died
fatal: bad object bf4086161ade363a3a851805690f5babb24e9914
fatal: remote did not send all necessary objects

Comment 5 Oonkwee Lim_ 2018-01-31 03:27:05 UTC
Works with write-behind:

[root@v3317a mnt]# gluster v info tprep3-vol
 
Volume Name: tprep3-vol
Type: Replicate
Volume ID: d70f3972-e48e-4fb9-b8fa-39a64190bd0c
Status: Started
Snapshot Count: 2
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: v3317a:/rhs/brick2/threp3brick
Brick2: v3317b:/rhs/brick2/threp3brick
Brick3: v3317c:/rhs/brick2/threp3brick
Options Reconfigured:
nfs.disable: on

[root@v3317a mnt]#  git clone --bare --mirror /tmp/ansible.bundle temp
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
fatal: premature end of pack file, 28 bytes missing
error: index-pack died
fatal: bad object bf4086161ade363a3a851805690f5babb24e9914
fatal: remote did not send all necessary objects

[root@v3317a mnt]# gluster volume set tprep3-vol performance.write-behind on
volume set: success

[root@v3317a mnt]#  git clone --bare --mirror /tmp/ansible.bundle temp
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
fatal: premature end of pack file, 28 bytes missing
error: index-pack died
fatal: bad object bf4086161ade363a3a851805690f5babb24e9914
fatal: remote did not send all necessary objects

[root@v3317a mnt]# gluster volume set tprep3-vol performance.write-behind off
volume set: success

[root@v3317a mnt]#  git clone --bare --mirror /tmp/ansible.bundle temp
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.

[root@v3317a mnt]#  git clone --bare --mirror /tmp/ansible.bundle temp
fatal: destination path 'temp' already exists and is not an empty directory.

[root@v3317a mnt]# rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.

[root@v3317a mnt]# rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.

=-=-=-=-= With the while loop =-=-=-=-=
[root@v3317a mnt]# while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
^C

Comment 13 Vivek Das 2018-05-06 08:41:52 UTC
while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (3/3), done.
Cloning into bare repository 'temp'...
Receiving objects: 100% (17/17), done.



Validated this bug as per the reproducer steps mentioned above. The issue is resolved. Marking this verified against the below version.

Version:
---------------
glusterfs-server-3.12.2-8.el7rhgs.x86_64

Comment 15 errata-xmlrpc 2018-09-04 06:42:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607


Note You need to log in before you can comment on or make changes to this bug.