Bug 1267579 - Gitlab repositories with disabled or missing filelists.xml database cannot be synced. However tools like yum/reposync are working
Summary: Gitlab repositories with disabled or missing filelists.xml database cannot be...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.1.1
Hardware: Unspecified
OS: Linux
medium
medium
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Daniel Lobato Garcia
URL:
Whiteboard:
: 1294196 1337662 (view as bug list)
Depends On:
Blocks: 1122832
TreeView+ depends on / blocked
 
Reported: 2015-09-30 13:08 UTC by Marcel Gazdík
Modified: 2023-09-14 23:58 UTC (History)
27 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-21 16:54:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Warning message when syncing Gitlab CE repo (44.33 KB, image/png)
2017-08-30 09:34 UTC, Daniel Lobato Garcia
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 1287 0 High CLOSED - CURRENTRELEASE Repo sync failing with KeyError 2017-01-16 21:04:31 UTC
Red Hat Knowledge Base (Solution) 2108001 0 None None None 2017-04-04 13:58:18 UTC

Comment 4 pulp-infra@redhat.com 2015-10-13 21:00:23 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2015-10-13 21:00:25 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 9 Bryan Kearney 2015-10-16 20:01:01 UTC
Based on these comments, closing this as wontfix. The git repos are malformed.

Comment 11 pulp-infra@redhat.com 2015-10-19 15:39:42 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 12 jean-pierre.pitout 2016-05-17 19:39:02 UTC
Hi,

Couldn't help notice that GitLab have a "sslcacert" line at the end of their repo config but no place for this configuration in Satellite 6:

[gitlab_gitlab-ce]
name=gitlab_gitlab-ce
baseurl=https://packages.gitlab.com/gitlab/gitlab-ce/el/7/$basearch
repo_gpgcheck=1
gpgcheck=0
enabled=1
gpgkey=https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

My question, is this why the metadata cannot be retrieved by Satellite OR are the metdata/repos malformed?

Regards,
JP Pitout

Comment 13 jean-pierre.pitout 2016-05-17 19:42:48 UTC
Could installing this cert in a keystore somewhere aleviate this problem?

Comment 14 Ina Panova 2016-05-19 16:09:54 UTC
(In reply to jean-pierre.pitout from comment #13)
> Could installing this cert in a keystore somewhere aleviate this problem?

Hello, the gitlab repositories are malformed, the cert will not help you. For more details of this issue you could check upstream bug.

Comment 15 Ian Koenig 2016-06-08 16:48:45 UTC
How are they malformed?   What is the definition of a well formed yum repo?   I can't find a public definition anywhere.  

If this is malformed why does yum work?

Comment 16 Ina Panova 2016-06-09 09:11:17 UTC
(In reply to Ian Koenig from comment #15)
> How are they malformed?   What is the definition of a well formed yum repo? 
> I can't find a public definition anywhere.  
> 
> If this is malformed why does yum work?

feel free to check upstream bug for more info https://pulp.plan.io/issues/1287#note-9

Comment 17 Marcel Gazdík 2016-06-09 15:01:28 UTC
@Ina: Hello, I believe that we should at least tell this to the user while the sync is in progress. As far as I know, it can be checked if the metadata are correct almost in the beginning of the complete process. 
  Since there are downloaded two files - other.db and filelist.db which is the corrupted one. The only step what should be done is checking the integrity, if the filelist.db provided the same list of keys as the other.db (package names if I remember it correctly). And if not, then there should be thrown some exception which will state this instead of letting the process go till it reach the end where all the downloaded (working) data are just removed and everything is pretending that it went great.

Comment 18 Ina Panova 2016-06-10 09:06:55 UTC
(In reply to Marcel Gazdík from comment #17)
> @Ina: Hello, I believe that we should at least tell this to the user while
> the sync is in progress. As far as I know, it can be checked if the metadata
> are correct almost in the beginning of the complete process. 
>   Since there are downloaded two files - other.db and filelist.db which is
> the corrupted one. The only step what should be done is checking the
> integrity, if the filelist.db provided the same list of keys as the other.db
> (package names if I remember it correctly). And if not, then there should be
> thrown some exception which will state this instead of letting the process
> go till it reach the end where all the downloaded (working) data are just
> removed and everything is pretending that it went great.

Agree, the purpose of the upstream bugfix would be to fail gracefully with some meaningful message.

Comment 19 Brian Bouterse 2016-11-21 18:39:06 UTC
*** Bug 1294196 has been marked as a duplicate of this bug. ***

Comment 20 Michael Hrivnak 2016-12-12 13:47:28 UTC
*** Bug 1337662 has been marked as a duplicate of this bug. ***

Comment 22 pulp-infra@redhat.com 2016-12-12 15:33:27 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 23 pulp-infra@redhat.com 2016-12-14 11:34:11 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 24 pulp-infra@redhat.com 2016-12-16 14:32:48 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 25 pulp-infra@redhat.com 2016-12-16 15:02:47 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 26 pulp-infra@redhat.com 2017-01-10 02:04:15 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 27 pulp-infra@redhat.com 2017-01-16 21:04:32 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 33 Daniel Lobato Garcia 2017-08-30 09:33:29 UTC
Verified.

Tested with Satellite 6.3 - Snap 13

Check the attached Screenshot, the error about malformed metadata is reported during syncing instead of the original confusing behavior. This was the goal of the Pulp developers as far as I can tell from this thread and https://pulp.plan.io/issues/1287#note-9

It looks like there is no intention from Pulp's side of working around malformed repositories like these, so I'm going to go with verify with the solution proposed, and mark Ina/mhrivnak as NEEDINFO: please answer if there is any intention of working around such repos, and if so, open a BZ o link to it.

Thanks.

Comment 34 Daniel Lobato Garcia 2017-08-30 09:34:46 UTC
Created attachment 1319967 [details]
Warning message when syncing Gitlab CE repo

Comment 35 Ina Panova 2017-08-30 10:35:27 UTC
we do not plan to workaround this kind of malformed repos.

Comment 37 Satellite Program 2018-02-21 16:54:17 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336

Comment 38 Red Hat Bugzilla 2023-09-14 23:58:33 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days


Note You need to log in before you can comment on or make changes to this bug.