Bug 1267579 - Gitlab repositories with disabled or missing filelists.xml database cannot be synced. However tools like yum/reposync are working
Gitlab repositories with disabled or missing filelists.xml database cannot be...
Status: VERIFIED
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp (Show other bugs)
6.1.1
Unspecified Linux
medium Severity medium (vote)
: GA
: --
Assigned To: satellite6-bugs
Daniel Lobato Garcia
: Reopened
: 1294196 1337662 (view as bug list)
Depends On:
Blocks: 1122832
  Show dependency treegraph
 
Reported: 2015-09-30 09:08 EDT by Marcel Gazdík
Modified: 2017-11-13 14:20 EST (History)
22 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-16 16:01:01 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Warning message when syncing Gitlab CE repo (44.33 KB, image/png)
2017-08-30 05:34 EDT, Daniel Lobato Garcia
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2108001 None None None 2017-04-04 09:58 EDT
Pulp Redmine 1287 High CLOSED - CURRENTRELEASE Repo sync failing with KeyError 2017-01-16 16:04 EST

  None (edit)
Comment 4 pulp-infra@redhat.com 2015-10-13 17:00:23 EDT
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 5 pulp-infra@redhat.com 2015-10-13 17:00:25 EDT
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 9 Bryan Kearney 2015-10-16 16:01:01 EDT
Based on these comments, closing this as wontfix. The git repos are malformed.
Comment 11 pulp-infra@redhat.com 2015-10-19 11:39:42 EDT
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 12 jean-pierre.pitout 2016-05-17 15:39:02 EDT
Hi,

Couldn't help notice that GitLab have a "sslcacert" line at the end of their repo config but no place for this configuration in Satellite 6:

[gitlab_gitlab-ce]
name=gitlab_gitlab-ce
baseurl=https://packages.gitlab.com/gitlab/gitlab-ce/el/7/$basearch
repo_gpgcheck=1
gpgcheck=0
enabled=1
gpgkey=https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

My question, is this why the metadata cannot be retrieved by Satellite OR are the metdata/repos malformed?

Regards,
JP Pitout
Comment 13 jean-pierre.pitout 2016-05-17 15:42:48 EDT
Could installing this cert in a keystore somewhere aleviate this problem?
Comment 14 Ina Panova 2016-05-19 12:09:54 EDT
(In reply to jean-pierre.pitout from comment #13)
> Could installing this cert in a keystore somewhere aleviate this problem?

Hello, the gitlab repositories are malformed, the cert will not help you. For more details of this issue you could check upstream bug.
Comment 15 Ian Koenig 2016-06-08 12:48:45 EDT
How are they malformed?   What is the definition of a well formed yum repo?   I can't find a public definition anywhere.  

If this is malformed why does yum work?
Comment 16 Ina Panova 2016-06-09 05:11:17 EDT
(In reply to Ian Koenig from comment #15)
> How are they malformed?   What is the definition of a well formed yum repo? 
> I can't find a public definition anywhere.  
> 
> If this is malformed why does yum work?

feel free to check upstream bug for more info https://pulp.plan.io/issues/1287#note-9
Comment 17 Marcel Gazdík 2016-06-09 11:01:28 EDT
@Ina: Hello, I believe that we should at least tell this to the user while the sync is in progress. As far as I know, it can be checked if the metadata are correct almost in the beginning of the complete process. 
  Since there are downloaded two files - other.db and filelist.db which is the corrupted one. The only step what should be done is checking the integrity, if the filelist.db provided the same list of keys as the other.db (package names if I remember it correctly). And if not, then there should be thrown some exception which will state this instead of letting the process go till it reach the end where all the downloaded (working) data are just removed and everything is pretending that it went great.
Comment 18 Ina Panova 2016-06-10 05:06:55 EDT
(In reply to Marcel Gazdík from comment #17)
> @Ina: Hello, I believe that we should at least tell this to the user while
> the sync is in progress. As far as I know, it can be checked if the metadata
> are correct almost in the beginning of the complete process. 
>   Since there are downloaded two files - other.db and filelist.db which is
> the corrupted one. The only step what should be done is checking the
> integrity, if the filelist.db provided the same list of keys as the other.db
> (package names if I remember it correctly). And if not, then there should be
> thrown some exception which will state this instead of letting the process
> go till it reach the end where all the downloaded (working) data are just
> removed and everything is pretending that it went great.

Agree, the purpose of the upstream bugfix would be to fail gracefully with some meaningful message.
Comment 19 Brian Bouterse 2016-11-21 13:39:06 EST
*** Bug 1294196 has been marked as a duplicate of this bug. ***
Comment 20 Michael Hrivnak 2016-12-12 08:47:28 EST
*** Bug 1337662 has been marked as a duplicate of this bug. ***
Comment 22 pulp-infra@redhat.com 2016-12-12 10:33:27 EST
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 23 pulp-infra@redhat.com 2016-12-14 06:34:11 EST
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 24 pulp-infra@redhat.com 2016-12-16 09:32:48 EST
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 25 pulp-infra@redhat.com 2016-12-16 10:02:47 EST
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Comment 26 pulp-infra@redhat.com 2017-01-09 21:04:15 EST
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 27 pulp-infra@redhat.com 2017-01-16 16:04:32 EST
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 33 Daniel Lobato Garcia 2017-08-30 05:33:29 EDT
Verified.

Tested with Satellite 6.3 - Snap 13

Check the attached Screenshot, the error about malformed metadata is reported during syncing instead of the original confusing behavior. This was the goal of the Pulp developers as far as I can tell from this thread and https://pulp.plan.io/issues/1287#note-9

It looks like there is no intention from Pulp's side of working around malformed repositories like these, so I'm going to go with verify with the solution proposed, and mark Ina/mhrivnak as NEEDINFO: please answer if there is any intention of working around such repos, and if so, open a BZ o link to it.

Thanks.
Comment 34 Daniel Lobato Garcia 2017-08-30 05:34 EDT
Created attachment 1319967 [details]
Warning message when syncing Gitlab CE repo
Comment 35 Ina Panova 2017-08-30 06:35:27 EDT
we do not plan to workaround this kind of malformed repos.

Note You need to log in before you can comment on or make changes to this bug.