Bug 2042730

Summary: Issues downloading RHEL 9 Beta packages from repo
Product: Red Hat Satellite Reporter: thattguy24601
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED NOTABUG QA Contact: Lai <ltran>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.10.1CC: dalley, lzap
Target Milestone: 6.11.0Keywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-02-02 19:37:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description thattguy24601 2022-01-20 03:08:14 UTC
Description of problem:

Kickstarting the RHEL 9 beta from Red Hat repositories gives errors when attempting to download certain packages.

Example message from failed kickstart: WRN packaging: Failed to download ‘shim-x64-15-16.el8.x86_64.rpm’: 3 - Interrupted by header callback: Server reports Content-Length: 682476 but expected size is: 683844

This happens with other packages if the system is set to BIOS instead of EFI, I saw the same message for python3-systemd-234-18.el9.x86_64.rpm

Additionally, if I change the sync policies of those repos from on-demand to immediate I see warnings on the sync about checksums from Red Hat's cdn not matching. Verifying the checksums doesn't appear to make this go away.

I was discussing this in Foreman's community forums here: 

Version-Release number of selected component (if applicable):

Pulp3

How reproducible:


Steps to Reproduce:
1. Have an installation of Satellite 6.10
2. Sync RHEL 9 beta kickstart repositories
3. Attempt kickstart

Actual results:

WRN packaging: Failed to download ‘shim-x64-15-16.el8.x86_64.rpm’: 3 - Interrupted by header callback: Server reports Content-Length: 682476 but expected size is: 683844

Expected results:

Successful kickstart

Additional info:

Comment 2 Daniel Alley 2022-01-26 05:02:30 UTC
@lzap or @thatguy, what is the specific URL that was used?

Comment 3 thattguy24601 2022-01-26 05:13:56 UTC
(In reply to Daniel Alley from comment #2)
> @lzap or @thatguy, what is the specific URL that was used?

Ultimately it's coming from https://cdn.redhat.com/content/beta/rhel9/9/x86_64/baseos/kickstart/Packages/shim-x64-16-16.el8.x86_64.rpm

Comment 8 Lukas Zapletal 2022-01-27 12:56:09 UTC
For the record, we have handed the problem over to the internal CDN team as it turned out there is a problem with the package or metadata. They have informed us that it should be now corrected.

What is the fix for downstream users then? Just re-syncing?

Thanks for help with this one.

Comment 9 Lukas Zapletal 2022-01-27 12:57:01 UTC
Apologies dralley for the misclick. I was asking the other Daniel. Cheers!

Comment 10 Daniel Alley 2022-01-27 14:05:20 UTC
That is also me but I'm now wondering how my school email address from 5 years ago got into the option list :)

Yes, just re-syncing should be enough.

Comment 11 thattguy24601 2022-01-28 06:08:09 UTC
(In reply to Daniel Alley from comment #10)
> That is also me but I'm now wondering how my school email address from 5
> years ago got into the option list :)
> 
> Yes, just re-syncing should be enough.

Looks like it's moving in the right direction, but now I see similar errors on other packages:

For example I now see the same error on shim-ia32-15-16.el8.x86_64. rpm on the BaseOS kickstart repo and python3-dns-2.1.0-6.el9.noarch.rpm from the AppStream kickstart repo. I suspect there may be others, these are just the first ones to throw an error

Comment 12 Daniel Alley 2022-01-28 14:35:41 UTC
Apologies @thatguy

The distribution team is going to do an audit / refresh to get those packages replaced, we'll let you know when that is finished (and run a full sync to verify ourselves).

Comment 13 Daniel Alley 2022-02-01 19:36:04 UTC
Looks like the updates are starting to flow through the system, will notify you when things seem to be fully updated.  The TL;DR on this issue is that the RHEL 9 beta kickstart repos were slightly out of sync with the standard RHEL 9 beta repos due to an oversight in some processes that only applied to the beta.

Comment 14 Daniel Alley 2022-02-01 22:30:45 UTC
I was able to do a successful immediate-mode sync of the repo, it should now be resolved.

Comment 15 thattguy24601 2022-02-02 01:03:56 UTC
(In reply to Daniel Alley from comment #14)
> I was able to do a successful immediate-mode sync of the repo, it should now
> be resolved.

I can confirm BaseOS is working as expected

Still running into a checksum issue on AppStream, now the package it's complaining about is CUnit-2.1.3-25.el9.x86_64.rpm

Comment 16 Daniel Alley 2022-02-02 01:09:18 UTC
The updates to that repo might still be rolling out but I'll double check with them.

Comment 17 Daniel Alley 2022-02-02 15:05:47 UTC
It was a caching issue, the issue was "fixed" but the caches hadn't been flushed yet and it was still serving the old RPM.

Now though, I am able to sync AppStream successfully. Thanks for your patience!

Comment 18 thattguy24601 2022-02-02 19:35:06 UTC
(In reply to Daniel Alley from comment #17)
> It was a caching issue, the issue was "fixed" but the caches hadn't been
> flushed yet and it was still serving the old RPM.
> 
> Now though, I am able to sync AppStream successfully. Thanks for your
> patience!

Of course,it is beta after all. I can confirm that I was able to successfully sync as well. Thanks for all your help with this

Comment 19 thattguy24601 2022-04-11 16:22:58 UTC
Hey, I just wanted to follow up and say I'm seeing some of the same issue in the non-kickstart AppStream repo, there were a few different packages and I don't remember all of them but gitk was definitely one of them

Comment 20 Daniel Alley 2022-04-11 23:30:30 UTC
I was able to sync https://cdn.redhat.com/content/beta/rhel9/9/x86_64/appstream/os/ successfully.  Is this the repo URL you're referring to or a different one?

Comment 21 thattguy24601 2022-04-12 01:03:33 UTC
(In reply to Daniel Alley from comment #20)
> I was able to sync
> https://cdn.redhat.com/content/beta/rhel9/9/x86_64/appstream/os/
> successfully.  Is this the repo URL you're referring to or a different one?

Yes, that the URL I'm referring to

Comment 22 Daniel Alley 2022-04-12 01:59:27 UTC
Can you still reproduce the issue if you try again?

Comment 23 thattguy24601 2022-04-12 02:22:24 UTC
(In reply to Daniel Alley from comment #22)
> Can you still reproduce the issue if you try again?

I can. Not when syncing, but when trying to do a dnf install.

I tried to dnf install gitk and got the following message:

[MIRROR] tk-6.8.10-8.el9.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 1704590 but expected size is: 1704086

Comment 24 Daniel Alley 2022-04-12 04:05:41 UTC
The log line says it's the tk package not the gitk package, but gitk pulls in "tk" as a dependency so that still makes sense. 

Could you download the repomd.xml and $$$-primary.xml metadata your Satellite is distributing for that repo and attach it?

Also which Satellite release are you on presently, 6.10.4?

Comment 25 thattguy24601 2022-04-12 04:23:29 UTC
(In reply to Daniel Alley from comment #24)
> The log line says it's the tk package not the gitk package, but gitk pulls
> in "tk" as a dependency so that still makes sense. 
> 
> Could you download the repomd.xml and $$$-primary.xml metadata your
> Satellite is distributing for that repo and attach it?
> 
> Also which Satellite release are you on presently, 6.10.4?

Actually haven't updated in a while so it's Satellite 6.10.1. Not sure the best way to capture repomd.xml and $$$-primary.xml

Comment 26 Daniel Alley 2022-05-18 15:12:48 UTC
@thattguy24601 Have you tried with a newer version and/or the official RHEL 9 release?

Comment 27 Red Hat Bugzilla 2023-09-15 01:51:13 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days