RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1833074 - reposync --newest-only does not download the latest package
Summary: reposync --newest-only does not download the latest package
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: yum
Version: 8.2
Hardware: All
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 8.0
Assignee: Marek Blaha
QA Contact: Luca Berton
URL:
Whiteboard:
Depends On: 1783238 1783244
Blocks: 1870265
TreeView+ depends on / blocked
 
Reported: 2020-05-07 18:26 UTC by jcastran
Modified: 2023-12-15 17:51 UTC (History)
23 users (show)

Fixed In Version: dnf-plugins-core-4.0.17-5.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1870265 (view as bug list)
Environment:
Last Closed: 2020-11-04 01:53:12 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1775434 0 medium CLOSED reposync does not properly sync modules with --newest option 2024-03-25 15:37:28 UTC
Red Hat Knowledge Base (Solution) 5186621 0 None None None 2020-09-25 13:33:16 UTC
Red Hat Product Errata RHEA-2020:4510 0 None None None 2020-11-04 01:53:31 UTC

Description jcastran 2020-05-07 18:26:59 UTC 
Description of problem:
reposync seems to download the "newest" based on alphabetical sorting and not based on the rpm versions. Because we can not create module.yaml repodata, we need to use --download-metadata. This means that the repository knows about every package, but only has the latest versions downloaded. When you then try to download the newest package, yum finds in the repodata the real latest version, and can't download it because it's not there. 

rhnsd is one example of this and the reason I assumed it's alphabetical or sorting with the same logic as `ls -l` instead of `yum list`


Version-Release number of selected component (if applicable):
 yum-utils-4.0.12-3.el8.noarch
 dnf-4.2.17-6.el8.noarch

How reproducible:
 Everytime

Steps to Reproduce:
1. mkdir /repo
2. reposync -n --repo rhel-8-for-x86_64-appstream-rpms --download-metadata -p /repo
3. vim /etc/yum.repos.d/my.repo
[local-appstream]
name=local sync of appstream
baseurl=file:///repo/rhel-8-for-x86_64-appstream-rpms
enabled=1
4. yumdownloader rhnsd --disablerepo="*" --enablerepo=local-appstream

Actual results:
  Error opening file for checksum: /repo/rhel-8-for-x86_64-appstream-rpms/Packages/r/rhnsd-5.0.35-3.module+el8+2754+6a08e8f4.x86_64.rpm
  Package "rhnsd-5.0.35-3.module+el8+2754+6a08e8f4.x86_64" from local repository "appstream" has incorrect checksum
  Error: Some packages from local repository have incorrect checksum
 
Expected results:
  rhnsd-5.0.35-3.module+el8+2754+6a08e8f4.x86_64.rpm           86 kB/s |  51 kB     00:00

Additional info:
Reposync downloaded this version which it considered the newest. 
# ls /repo/rhel-8-for-x86_64-appstream-rpms/Packages/r/rhnsd-*
/repo/rhel-8-for-x86_64-appstream-rpms/Packages/r/rhnsd-5.0.35-3.module+el8.1.0+3455+3ddf2832.x86_64.rpm

ls -l sorts the highest first. 
   rhnsd-5.0.35-3.module+el8.1.0+3455+3ddf2832.x86_64.rpm
   rhnsd-5.0.35-3.module+el8+2754+6a08e8f4.x86_64.rpm

But yum list shows the latest package at the bottom like this:
   rhnsd.x86_64            5.0.35-3.module+el8.1.0+3455+3ddf2832               appstream
   rhnsd.x86_64            5.0.35-3.module+el8+2754+6a08e8f4                   appstream

This is seemingly the difference in why it downloaded the wrong package.
Comment 8 Leon Kang 2020-07-16 15:29:50 UTC 
Hey Marek, i'm trying to understand the problem as you have commented above if you could help to correct my understanding:

"The problem here is that newer module versions contains packages with older NEVRAs" - Sounds like we will need to adjust this so the older ones are not showing up
"So in the end the newest rhnsd package (by NEVRA - that is what yumdownloader is trying to download) is contained in older version of the module (which was not downloaded by reposync)."
- rhnsd-0:5.0.35-3.module+el8+2754+6a08e8f4.x86_64 is contained in older version rhnsd-0:5.0.35-3.module+el8.0.0.z+3494+e6f475f6.x86_64 but this older module is not downloaded causing this issue.

Is this understanding correct?
Comment 9 Marek Blaha 2020-07-21 07:54:06 UTC 
The problem here is that `reposync -n` and `yum download` use different algorithms to recognize what is latest:

- reposync downloads artifacts from the latest module version (so it downloads rhnsd-0:5.0.35-3.module+el8.1.0+3455+3ddf2832.x86_64 from satellite-5-client module version 8010020190621091459)

- on the other hand `yum download` downloads the package with the highest NEVRA available (rhnsd-5.0.35-3.module+el8+2754+6a08e8f4.x86_64.rpm because this NEVRA is after the `+el8+ -> +el.8.Y.Z+` change in the release number scheme the highest one).

And it is not only the download command. Also `install` command will try to "upgrade" the rhnsd package:

# dnf install rhnsd
Updating Subscription Management repositories.
Last metadata expiration check: 0:20:26 ago on Tue Jul 21 03:28:52 2020.
Package rhnsd-5.0.35-3.module+el8.1.0+3455+3ddf2832.x86_64 is already installed.
Dependencies resolved.
======================================================================================================
 Package  Arch      Version                                 Repository                           Size
======================================================================================================
Upgrading:
 rhnsd    x86_64    5.0.35-3.module+el8+2754+6a08e8f4       rhel-8-for-x86_64-appstream-rpms     51 k

Transaction Summary
======================================================================================================
Upgrade  1 Package
Comment 13 jcastran 2020-07-22 11:57:50 UTC 
There are 2 workarounds for this issue.

1. Do not use the -n (--newest) flag. This will result in the download of every package in that repository and an increased storage requirement. The full size can be seen with "yum repolist -v"

  Repo-id            : rhel-8-for-x86_64-appstream-rpms
  Repo-size          : 25 G

  Repo-id            : rhel-8-for-x86_64-baseos-rpms
  Repo-size          : 6.2 G

2. After using "reposync -n --download-metadata --repo=<repo-id>", when a package is identified, you can download the latest and put it in the repository with no further changes. The repodata already thinks the package is there. Using rhsnd as my example.

    # yumdownloader rhnsd
    # mv rhnsd-5.0.35-3.module+el8+2754+6a08e8f4.x86_64.rpm <repo-id>/Packages/r/
Comment 17 Marek Blaha 2020-07-28 09:21:27 UTC 
Here is PR with reposync fix: https://github.com/rpm-software-management/dnf-plugins-core/pull/407
Comment 41 errata-xmlrpc 2020-11-04 01:53:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (yum bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4510
Note You need to log in before you can comment on or make changes to this bug.