Bug 1538366
| Summary: | [GSS] Git clone --bare --mirror of git bundle fails when cloning on gluster storage | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Oonkwee Lim <olim> | ||||
| Component: | write-behind | Assignee: | Csaba Henk <csaba> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Vivek Das <vdas> | ||||
| Severity: | low | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | rhgs-3.2 | CC: | amukherj, atumball, msaini, rgowdapp, rhinduja, rhs-bugs, storage-qa-internal, vdas | ||||
| Target Milestone: | --- | ||||||
| Target Release: | RHGS 3.4.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | glusterfs-3.12.2-1 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-09-04 06:42:04 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1503138 | ||||||
| Attachments: |
|
||||||
Looks like the issue is similar to the other DB consistency issues we saw. Can it be tried with disable open-behind translator (and if that doesn't work, try with disabling other perf translators. Also, if its reproducible, can we add this as a test case in QE testing? Hello Amar, Reproducer steps: 1) create and start a replica 3 volume 2) copy ansible.bundle to /tmp 3) mount the volume onto /mnt 4) run while loop # while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done I am trying out your suggestions. Turn off open-behind did not help: [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle tempCloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects [root@v3317a mnt]# gluster volume set tprep3-vol open-behind off volume set: success [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects Works with write-behind: [root@v3317a mnt]# gluster v info tprep3-vol Volume Name: tprep3-vol Type: Replicate Volume ID: d70f3972-e48e-4fb9-b8fa-39a64190bd0c Status: Started Snapshot Count: 2 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: v3317a:/rhs/brick2/threp3brick Brick2: v3317b:/rhs/brick2/threp3brick Brick3: v3317c:/rhs/brick2/threp3brick Options Reconfigured: nfs.disable: on [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects [root@v3317a mnt]# gluster volume set tprep3-vol performance.write-behind on volume set: success [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects [root@v3317a mnt]# gluster volume set tprep3-vol performance.write-behind off volume set: success [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. [root@v3317a mnt]# git clone --bare --mirror /tmp/ansible.bundle temp fatal: destination path 'temp' already exists and is not an empty directory. [root@v3317a mnt]# rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. [root@v3317a mnt]# rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. =-=-=-=-= With the while loop =-=-=-=-= [root@v3317a mnt]# while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. ^C while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Validated this bug as per the reproducer steps mentioned above. The issue is resolved. Marking this verified against the below version. Version: --------------- glusterfs-server-3.12.2-8.el7rhgs.x86_64 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607 |
Created attachment 1385817 [details] the ansible bundle Issue : Git clone --bare --mirror of git bundle fails when cloning on gluster storage Issue description ------ The customer is encountering the following issue: $ git clone --bare --mirror ../ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects After a few retries it eventually succeeds: $ git clone --bare --mirror ../ansible.bundle temp Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Is reproducible ? Yes, below are my steps: Create a replica 3 volume. Copy the ansible.bundle to /tmp. Run a while loop: [root@v3317a mnt]# mount | grep glusterfs v3317a:/rep3-vol on /mnt type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) [root@v3317a mnt]# while :; do rm -rf temp; git clone --bare --mirror /tmp/ansible.bundle temp; sleep 1; done Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. fatal: premature end of pack file, 28 bytes missing error: index-pack died fatal: bad object bf4086161ade363a3a851805690f5babb24e9914 fatal: remote did not send all necessary objects Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. Cloning into bare repository 'temp'... Receiving objects: 100% (17/17), done. Resolving deltas: 100% (3/3), done. ^C =-=-=-=-= Customer's Environment details : GlusterFS version (from all nodes) Kernel version used (if required) I was able to reproduced this on my 3.3.1 async. I could not reproduce this on the local FS. I have attached the ansible bundle file that is causing grief.