Bug 1821182

Summary: [Repository] - Packages are not getting synced with rpm-with-sha-512
Product: Red Hat Satellite Reporter: Omkar Khatavkar <okhatavk>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Lai <ltran>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.7.0CC: bkearney, bmbouter, dkliban, egolov, ehelms, ggainey, ipanova, jacob.a.green, jsherril, kkinge, mawerner, mmccune, rchan, smallamp, ttereshc
Target Milestone: 6.8.0Keywords: Triaged
Target Release: Unused   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: pulp-rpm-2.21.2 Doc Type: Known Issue
Doc Text:
Using the "Immediate" download policy to synchronize RPM repositories that are signed with SHA-512 hashes causes the synchronization to fail with the following error: "KeyError: u'sha512'". To work around this issue, change the download policy to "On Demand" for repositories that are signed with SHA-512 hashes.
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 13:01:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Omkar Khatavkar 2020-04-06 09:10:38 UTC
Description of problem:
[Repository] - Packages are not getting copied over with rpm-with-sha-512

Version-Release number of selected component (if applicable):
exist in 6.7 snap 20 
Not exist in 6.6  

How reproducible:
always

Steps to Reproduce:
1.create a repository with url "https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-sha-512/"


Actual results:
packages are not getting copied in case if the repo is sha-512 

Expected results:
packages should get copied in case if the repo is sha-512 

Additional info:

Comment 3 Justin Sherrill 2020-04-06 14:05:53 UTC
I can reproduce on the latest 6.7: 

[root@sat-6-7-qa-rhel7 ~]# rpm -qa | grep pulp | sort
libmodulemd-1.7.0-1.pulp.el7sat.x86_64
libsolv-0.7.4-4.pulp.el7sat.x86_64
pulp-client-1.0-1.noarch
pulp-docker-plugins-3.2.5-1.el7sat.noarch
pulp-katello-1.0.3-1.el7sat.noarch
pulp-maintenance-2.21.0-1.el7sat.noarch
pulp-ostree-plugins-1.3.1-2.el7sat.noarch
pulp-puppet-plugins-2.21.0-1.el7sat.noarch
pulp-puppet-tools-2.21.0-1.el7sat.noarch
pulp-rpm-plugins-2.21.0.4-1.el7sat.noarch
pulp-selinux-2.21.0-1.el7sat.noarch
pulp-server-2.21.0-1.el7sat.noarch
python2-solv-0.7.4-4.pulp.el7sat.x86_64
python-pulp-common-2.21.0-1.el7sat.noarch
python-pulp-docker-common-3.2.5-1.el7sat.noarch
python-pulp-integrity-2.21.0.4-1.el7sat.noarch
python-pulp-oid_validation-2.21.0-1.el7sat.noarch
python-pulp-ostree-common-1.3.1-2.el7sat.noarch
python-pulp-puppet-common-2.21.0-1.el7sat.noarch
python-pulp-repoauth-2.21.0-1.el7sat.noarch
python-pulp-rpm-common-2.21.0.4-1.el7sat.noarch
python-pulp-streamer-2.21.0-1.el7sat.noarch



Pulp throws this in the logs:

Download succeeded: https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-sha-512/zebra-0.1-2.noarch.rpm.
ERROR: (8377-95616) Listener error on event: succeeded
ERROR: (8377-95616) Traceback (most recent call last):
ERROR: (8377-95616)   File "/usr/lib/python2.7/site-packages/pulp/server/content/sources/event.py", line 39, in __call__
ERROR: (8377-95616)     listener.on_event(self)
ERROR: (8377-95616)   File "/usr/lib/python2.7/site-packages/pulp/server/content/sources/event.py", line 135, in on_event
ERROR: (8377-95616)     method(event.request)
ERROR: (8377-95616)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/alternate.py", line 119, in on_succeeded
ERROR: (8377-95616)     self.content_listener.download_succeeded(report)
ERROR: (8377-95616)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/listener.py", line 195, in download_succeeded
ERROR: (8377-95616)     super(RPMListener, self).download_succeeded(report)
ERROR: (8377-95616)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/listener.py", line 86, in download_succeeded
ERROR: (8377-95616)     self._verify_checksum(unit, report.destination)
ERROR: (8377-95616)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/listener.py", line 150, in _verify_checksum
ERROR: (8377-95616)     if sums[unit.checksumtype] != unit.checksum:
ERROR: (8377-95616) KeyError: u'sha512'



Strangely, on an older 6.7 snap it was fine, and the only difference in pulp versions was pulp-rpm-plugins-2.21.0.4-1.el7sat.noarch  vs pulp-rpm-plugins-2.21.0.3-1.el7sat.noarch

Moving to the pulp component

Comment 4 Justin Sherrill 2020-04-06 14:07:10 UTC
Also, to clarify, the full reproducer steps:


Steps to Reproduce:
1. create a repository with url "https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-sha-512/"
2. sync the repository
3. check the rpm count in the repo

Comment 6 Justin Sherrill 2020-04-06 16:34:10 UTC
After some investigation tanya and I discovered:

* This is not a regression, the issue happens on 6.6 if you create the custom repository's download_policy as 'immediate'
* In 6.7, the default download policy for custom repos was changed from on_demand to immediate
* Any upgrade from 6.6 will keep that default of 'on_demand', hence why i didn't see the error on an upgraded satellite
* This means that likely on_demand repositories are NOT having their content validated upon download, but this may need some more investigation 

But in summary:

This is not a regression, and should not break any existing users that are upgrading to satellite 6.7.  Users that do a fresh install of satellite 6.7 can workaround the issue by switching to on demand for their download policy.

Comment 7 Sudhir Mallamprabhakara 2020-04-06 16:46:40 UTC
Omkar, Please test and confirm.

-Sudhir

Comment 8 Omkar Khatavkar 2020-04-07 06:05:05 UTC
Not a regression issue as this exists in Satellite 6.6 with if Download Policy is Immediate. Making it them to on-demand workaround works. As Default Custom Repository download policy did not exist earlier version in 6.6 and making default value as 'Immediate' causing issue.

Comment 9 Mike McCune 2020-04-07 15:12:20 UTC
We are moving this out of 6.7 as it is not a regression. From Justin Sherrill:

"""
After some investigation Tanya and I discovered:

* This is not a regression, the issue happens on 6.6 if you create the custom repository's download_policy as 'immediate'
* In 6.7, the default download policy for custom repos was changed from on_demand to immediate
* Any upgrade from 6.6 will keep that default of 'on_demand', hence why i didn't see the error on an upgraded satellite
* This means that likely on_demand repositories are NOT having their content validated upon download, but this may need some more investigation

But in summary:

This is not a regression, and should not break any existing users that are upgrading to satellite 6.7.  Users that do a fresh install of Satellite 6.7 can workaround the issue by switching to on demand for their download policy.
""

Comment 10 pulp-infra@redhat.com 2020-05-08 19:30:35 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2020-05-08 19:30:36 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 12 pulp-infra@redhat.com 2020-05-08 22:14:08 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 15 pulp-infra@redhat.com 2020-05-19 14:31:48 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 16 pulp-infra@redhat.com 2020-05-19 16:32:38 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 22 errata-xmlrpc 2020-10-27 13:01:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366