Bug 1400722 - [RFE] On find_conflicts check when running against debug repo downloads large number of files
Summary: [RFE] On find_conflicts check when running against debug repo downloads large...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: rpmdeplint
Classification: Community
Component: general
Version: unreleased
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: 1.3
Assignee: Tyrone Abdy
QA Contact: tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-01 23:38 UTC by Tyrone Abdy
Modified: 2017-04-28 05:31 UTC (History)
5 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-04-28 05:31:07 UTC
Embargoed:


Attachments (Terms of Use)

Description Tyrone Abdy 2016-12-01 23:38:35 UTC
Description of problem:

If you run it against a debug repository every debuginfo rpm has the file /usr/lib/debug which will then proceed to download every file within find_conflicts to check whether the conflict is correct or not.

Some additional logic could be added that would reduce the rpms that are needed to download, the current idea is that for more than one conflicts on a single file to assume if that the package under test doesn't conflict on that file with the first package (or does) that should be enough to report a conflict without the need to download potentially thousands of packages.


How reproducible:

100%

Steps to Reproduce:
1. run rpmdeplint check against debug repo with --debug on
 e.g /content/dist/rhel/server/6/6Server/x86_64/debug/ with a debuginfo rpm.


Actual results:

Takes a long time (8+hours) to download and check the file conflict of every package.

Expected results:

Reduced time for skipping long process of downloading packages.

Comment 2 Tyrone Abdy 2016-12-05 03:44:16 UTC
https://gerrit.beaker-project.org/#/c/5496/

Comment 3 Tyrone Abdy 2016-12-05 03:45:32 UTC
So the current solution that I have went with in the above patch is to skip after the first conflict attempt and notify if there are more potential problems that have been noticed.

This currently works on the assumption that the first result (good or bad) is indicative of the result against the entire repository.

Comment 5 Dan Callaghan 2016-12-06 00:16:16 UTC
This problem was most noticeable with debuginfo packages, since they all share ownership of /usr/lib/debug, but there are other similar cases too (many packages own /usr/share/bash-completion for example) so I think it's still worth fixing.

Comment 6 Tyrone Abdy 2016-12-06 02:59:23 UTC
Considering we do actually run currently against debuginfo/debug repositories (not srpms), so at the moment within rpm-test-trigger you'll get a really long test run anyhow.

But maybe we should be doing some more filtering within rpm-test-trigger to not run against debug. But I am sure there will be other cases where there are large amounts of shared conflicts.

Comment 7 Tyrone Abdy 2016-12-07 03:14:06 UTC
After discussion on the first iteration it appears that is not sufficient based on the assumptions.

We considered using the filelists within a repo, but as they only provide whether a file is a directory or not, without considering the mode and owner of the file/directory which will not be sufficient to determine the whether the files or directories conflict.

So another attempt will be looking at creating a white list of common conflicts that can occur, so I'll investigate a few repositories to see the most common to get an idea of how big a white list could be.

Comment 8 Tyrone Abdy 2016-12-07 04:13:28 UTC
After running some stats on a few repos quickly. Here the amount of common files within a few repos, with any combinations of addons/optional/layered products.

Fedora 24 (system) 21.

RHEL 6 Resilient Storage Debug 14.

RHEL 6 Server 83.

RHEL 6 Server Debug 330~.

RHEL 7 Workstation 93.

So by this list it seems that there will be excluding the debug repositories, somewhere around 100 (or even more?) depending on the repository combinations as well of common folders/files.

In a white list, which would need to be maintained, with a script and updates as time goes on.

Comment 9 Dan Callaghan 2017-04-28 05:31:07 UTC
rpmdeplint 1.3 has been released.


Note You need to log in before you can comment on or make changes to this bug.