Bug 1400722

Summary:	[RFE] On find_conflicts check when running against debug repo downloads large number of files
Product:	[Community] rpmdeplint	Reporter:	Tyrone Abdy <tabdy>
Component:	general	Assignee:	Tyrone Abdy <tabdy>
Status:	CLOSED CURRENTRELEASE	QA Contact:	tools-bugs <tools-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	unreleased	CC:	bmcivor, dcallagh, jorris, rjoost, tabdy
Target Milestone:	1.3
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-04-28 05:31:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Tyrone Abdy 2016-12-01 23:38:35 UTC

Description of problem:

If you run it against a debug repository every debuginfo rpm has the file /usr/lib/debug which will then proceed to download every file within find_conflicts to check whether the conflict is correct or not.

Some additional logic could be added that would reduce the rpms that are needed to download, the current idea is that for more than one conflicts on a single file to assume if that the package under test doesn't conflict on that file with the first package (or does) that should be enough to report a conflict without the need to download potentially thousands of packages.


How reproducible:

100%

Steps to Reproduce:
1. run rpmdeplint check against debug repo with --debug on
 e.g /content/dist/rhel/server/6/6Server/x86_64/debug/ with a debuginfo rpm.


Actual results:

Takes a long time (8+hours) to download and check the file conflict of every package.

Expected results:

Reduced time for skipping long process of downloading packages.

Comment 2 Tyrone Abdy 2016-12-05 03:44:16 UTC

https://gerrit.beaker-project.org/#/c/5496/

Comment 3 Tyrone Abdy 2016-12-05 03:45:32 UTC

So the current solution that I have went with in the above patch is to skip after the first conflict attempt and notify if there are more potential problems that have been noticed.

This currently works on the assumption that the first result (good or bad) is indicative of the result against the entire repository.

Comment 5 Dan Callaghan 2016-12-06 00:16:16 UTC

This problem was most noticeable with debuginfo packages, since they all share ownership of /usr/lib/debug, but there are other similar cases too (many packages own /usr/share/bash-completion for example) so I think it's still worth fixing.

Comment 6 Tyrone Abdy 2016-12-06 02:59:23 UTC

Considering we do actually run currently against debuginfo/debug repositories (not srpms), so at the moment within rpm-test-trigger you'll get a really long test run anyhow.

But maybe we should be doing some more filtering within rpm-test-trigger to not run against debug. But I am sure there will be other cases where there are large amounts of shared conflicts.

Comment 7 Tyrone Abdy 2016-12-07 03:14:06 UTC

After discussion on the first iteration it appears that is not sufficient based on the assumptions.

We considered using the filelists within a repo, but as they only provide whether a file is a directory or not, without considering the mode and owner of the file/directory which will not be sufficient to determine the whether the files or directories conflict.

So another attempt will be looking at creating a white list of common conflicts that can occur, so I'll investigate a few repositories to see the most common to get an idea of how big a white list could be.

Comment 8 Tyrone Abdy 2016-12-07 04:13:28 UTC

After running some stats on a few repos quickly. Here the amount of common files within a few repos, with any combinations of addons/optional/layered products.

Fedora 24 (system) 21.

RHEL 6 Resilient Storage Debug 14.

RHEL 6 Server 83.

RHEL 6 Server Debug 330~.

RHEL 7 Workstation 93.

So by this list it seems that there will be excluding the debug repositories, somewhere around 100 (or even more?) depending on the repository combinations as well of common folders/files.

In a white list, which would need to be maintained, with a script and updates as time goes on.

Comment 9 Dan Callaghan 2017-04-28 05:31:07 UTC

rpmdeplint 1.3 has been released.