Bug 2116386

Summary: Possibly out-of-date CentOS Stream 8 AMIs
Product: Red Hat Enterprise Linux 8 Reporter: Frantisek Sumsal <fsumsal>
Component: distributionAssignee: RHEL Program Management <pm-rhel>
Status: CLOSED MIGRATED QA Contact: Release Test Team <release-test-team-automation>
Severity: medium Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: anoopcs, bstinson, egolov, jwboyer, pniahodk
Target Milestone: rcKeywords: MigratedToJIRA
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-14 13:24:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Frantisek Sumsal 2022-08-08 12:51:23 UTC 
Description of problem:
Recently in CentOS CI we started migration from on-prem baremetal machines to EC2 instances, both VMs and metal nodes for workloads which require access to KVM - where systemd being one of such workloads.

While migrating our systemd jobs over to metal machines, I stumbled across quite a conundrum: the metal machines can't be reasonably rebooted, since the reboot takes 10 - 20 minutes, so we have to "live" with the kernel provided by the latest AMI - in this case it's CentOS Stream 8. However, the kernel in the C8S AMI appears to be quite out of date:

# uname -r
4.18.0-305.3.1.el8.x86_64

Packages on one of the C8S mirrors:

[   ] kernel-4.18.0-305.10.2.el8_4.x86_64.rpm         2021-07-20 21:12  5.9M  
[   ] kernel-4.18.0-305.12.1.el8_4.x86_64.rpm         2021-08-11 05:46  5.9M  
[   ] kernel-4.18.0-305.17.1.el8_4.x86_64.rpm         2021-09-08 17:47  5.9M  
[   ] kernel-4.18.0-305.19.1.el8_4.x86_64.rpm         2021-09-15 20:50  5.9M  
[   ] kernel-4.18.0-305.25.1.el8_4.x86_64.rpm         2021-11-03 12:55  5.9M  
[   ] kernel-4.18.0-315.el8.x86_64.rpm                2021-06-28 22:45  6.4M  
[   ] kernel-4.18.0-326.el8.x86_64.rpm                2021-07-29 00:48  6.8M  
[   ] kernel-4.18.0-331.el8.x86_64.rpm                2021-08-19 20:23  6.9M  
[   ] kernel-4.18.0-338.el8.x86_64.rpm                2021-08-27 20:43  7.0M  
[   ] kernel-4.18.0-348.2.1.el8_5.x86_64.rpm          2021-11-17 17:09  7.0M  
[   ] kernel-4.18.0-348.7.1.el8_5.x86_64.rpm          2021-12-22 15:51  7.0M  
[   ] kernel-4.18.0-348.el8.x86_64.rpm                2021-10-20 18:52  7.0M  
[   ] kernel-4.18.0-358.el8.x86_64.rpm                2022-01-10 16:54  7.5M  
[   ] kernel-4.18.0-365.el8.x86_64.rpm                2022-02-10 18:54  7.7M  
[   ] kernel-4.18.0-373.el8.x86_64.rpm                2022-03-22 18:34  8.0M  
[   ] kernel-4.18.0-383.el8.x86_64.rpm                2022-04-20 22:10  8.1M  
[   ] kernel-4.18.0-394.el8.x86_64.rpm                2022-06-03 16:54  8.3M  
[   ] kernel-4.18.0-408.el8.x86_64.rpm                2022-07-19 16:50  8.5M  

This is quite unfortunate, since the sytemd-networkd test suite requires extra kernel modules from the kernel-modules-extra package, which is no longer available on mirrors for the kernel version used by the C8S AMI.

I'm not sure how often the CentOS Stream AMIs are supposed to be updated, but I guess it should be relatively often, given that CentOS Stream is basically a rolling release now?

Additional info:
Original CentOS Infra ticket: https://pagure.io/centos-infra/issue/873
Comment 1 Frantisek Sumsal 2022-08-08 13:34:11 UTC 
As pointed out by Fabian, there were apparently no new Cloud images for C8S released since January either - https://cloud.centos.org/centos/8-stream/x86_64/images/?C=M;O=D. However, for C9S they're generated regularly - https://cloud.centos.org/centos/9-stream/x86_64/images/?C=M;O=D.
Comment 2 Frantisek Sumsal 2022-09-13 07:56:11 UTC 
Any news there? The C8S AMIs are still very out of date[0] whereas the latest C9S one is ~two weeks old. This is really hindering our CentOS Stream testing (and by extension RHEL testing as well).

[0] https://cloud.centos.org/centos/8-stream/x86_64/images/?C=M;O=D
[1] https://cloud.centos.org/centos/9-stream/x86_64/images/?C=M;O=D
Comment 3 RHEL Program Management 2023-09-14 13:22:11 UTC 
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.
Comment 4 RHEL Program Management 2023-09-14 13:24:27 UTC 
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.