Bug 2138460

Summary: Unable to update ARM AMIs on AWS
Product: Red Hat Update Infrastructure for Cloud Providers Reporter: peninm
Component: OperationsAssignee: RHUI Bug List <rhui-bugs>
Status: CLOSED CURRENTRELEASE QA Contact: Radek Bíba <rbiba>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.0.1CC: adamkam, gtanzillo, gtanzill, jwarne, kyankovi, mminar, mreznik, srinivastp, tpfromme, vinprab
Target Milestone: ---Flags: peninm: needinfo? (tpfromme)
Target Release: ---   
Hardware: aarch64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-28 13:11:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description peninm 2022-10-28 22:49:26 UTC
Description of problem:
Unable to update ARM AMIs on AWS due to an issue with NetworkManager.


Version-Release number of selected component (if applicable):
package NetworkManager-cloud-setup-1:1.36.0-7.el8_6.aarch64 requires NetworkManager = 1:1.36.0-7.el8_6

How reproducible:
Happens every time

Steps to Reproduce:
1. Launch ARM (RHEL-8.6.0_HVM-20220503-arm64-2-Hourly2-GP2) version of RHEL 8 on AWS
2. Run `sudo dnf update`
3. Observe failure

Actual results:
Failure to update

Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:03:53 ago on Fri 28 Oct 2022 10:42:35 PM UTC.
Error:
 Problem 1: package NetworkManager-cloud-setup-1:1.36.0-7.el8_6.aarch64 requires NetworkManager = 1:1.36.0-7.el8_6, but none of the providers can be installed
  - cannot install both NetworkManager-1:1.36.0-9.el8_6.aarch64 and NetworkManager-1:1.36.0-7.el8_6.aarch64
  - cannot install the best update candidate for package NetworkManager-cloud-setup-1:1.36.0-4.el8.aarch64
  - cannot install the best update candidate for package NetworkManager-1:1.36.0-4.el8.aarch64
 Problem 2: problem with installed package NetworkManager-cloud-setup-1:1.36.0-4.el8.aarch64
  - package NetworkManager-cloud-setup-1:1.36.0-4.el8.aarch64 requires NetworkManager-libnm(aarch-64) = 1:1.36.0-4.el8, but none of the providers can be installed
  - package NetworkManager-cloud-setup-1:1.36.0-7.el8_6.aarch64 requires NetworkManager-libnm(aarch-64) = 1:1.36.0-7.el8_6, but none of the providers can be installed
  - cannot install both NetworkManager-libnm-1:1.36.0-9.el8_6.aarch64 and NetworkManager-libnm-1:1.36.0-4.el8.aarch64
  - cannot install both NetworkManager-libnm-1:1.36.0-9.el8_6.aarch64 and NetworkManager-libnm-1:1.36.0-7.el8_6.aarch64
  - cannot install the best update candidate for package NetworkManager-libnm-1:1.36.0-4.el8.aarch64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Expected results:
AMI installs latest update

Additional info:

Comment 1 Gregg Tanzillo 2022-10-29 04:18:59 UTC
I was able to recreate the issue, following the steps in the BZ (see output below). However, I don’t know if it’s a RHUi issue or possibly a problem with the package NetworkManager-cloud-setup-1:1.36.0-7.el8_6.aarch64, itself or the RHUI client on that image or an entitlement issue. 

I've emailed RHUI engineering asking for their assistance.

Comment 2 Gregg Tanzillo 2022-10-29 04:21:02 UTC
Here's my output:

[ec2-user@ip-172-32-32-42 ~]$ sudo dnf update
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Red Hat Enterprise Linux 8 for aarch64 - AppStream from RHUI (RPMs)                                                                                                                                                                        33 MB/s |  39 MB     00:01
Red Hat Enterprise Linux 8 for aarch64 - BaseOS from RHUI (RPMs)                                                                                                                                                                           41 MB/s |  42 MB     00:01
Red Hat Ansible Engine 2 for RHEL 8 (RPMs) from RHUI                                                                                                                                                                                      9.7 MB/s | 2.5 MB     00:00
RHUI Client Configuration Server 8                                                                                                                                                                                                         29 kB/s | 2.3 kB     00:00
Error:
 Problem 1: package NetworkManager-cloud-setup-1:1.36.0-7.el8_6.aarch64 requires NetworkManager = 1:1.36.0-7.el8_6, but none of the providers can be installed
  - cannot install both NetworkManager-1:1.36.0-9.el8_6.aarch64 and NetworkManager-1:1.36.0-7.el8_6.aarch64
  - cannot install the best update candidate for package NetworkManager-cloud-setup-1:1.36.0-4.el8.aarch64
  - cannot install the best update candidate for package NetworkManager-1:1.36.0-4.el8.aarch64
 Problem 2: problem with installed package NetworkManager-cloud-setup-1:1.36.0-4.el8.aarch64
  - package NetworkManager-cloud-setup-1:1.36.0-4.el8.aarch64 requires NetworkManager-libnm(aarch-64) = 1:1.36.0-4.el8, but none of the providers can be installed
  - package NetworkManager-cloud-setup-1:1.36.0-7.el8_6.aarch64 requires NetworkManager-libnm(aarch-64) = 1:1.36.0-7.el8_6, but none of the providers can be installed
  - cannot install both NetworkManager-libnm-1:1.36.0-9.el8_6.aarch64 and NetworkManager-libnm-1:1.36.0-4.el8.aarch64
  - cannot install both NetworkManager-libnm-1:1.36.0-9.el8_6.aarch64 and NetworkManager-libnm-1:1.36.0-7.el8_6.aarch64
  - cannot install the best update candidate for package NetworkManager-libnm-1:1.36.0-4.el8.aarch64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Comment 3 Radek Bíba 2022-10-29 07:34:58 UTC
Please try again.

Comment 4 Todd Pfrommer 2022-10-29 18:13:22 UTC
This is a Red Hat content issue. Red Hat RHUI ops has implemented a workaround in production and confirmed it resolves our reproducer. Please test the fix on the AWS side, @peninm

Comment 5 Srinivastp 2022-10-29 19:05:53 UTC
This issue is present in eu-central-1 region too.

[root@ec2-user]# yum check-update|grep NetworkManager
NetworkManager.aarch64             1:1.36.0-9.el8_6                            rhel-8-baseos-rhui-rpms
NetworkManager-cloud-setup.aarch64 1:1.36.0-7.el8_6                            rhel-8-appstream-rhui-rpms
NetworkManager-libnm.aarch64       1:1.36.0-9.el8_6                            rhel-8-baseos-rhui-rpms
NetworkManager-team.aarch64        1:1.36.0-9.el8_6                            rhel-8-baseos-rhui-rpms
NetworkManager-tui.aarch64         1:1.36.0-9.el8_6                            rhel-8-baseos-rhui-rpms
[root@ec2-user]# date
Sat Oct 29 18:56:32 UTC 2022
[root@ec2-user]#

[root@ec2-user]# curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document|grep region
  "region" : "eu-central-1",
[root@ec2-user]#
[root@ec2-user]# yum update
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:13:30 ago on Sat 29 Oct 2022 06:50:52 PM UTC.
Error:
 Problem 1: package NetworkManager-cloud-setup-1:1.36.0-7.el8_6.aarch64 requires NetworkManager = 1:1.36.0-7.el8_6, but none of the providers can be installed
  - cannot install both NetworkManager-1:1.36.0-9.el8_6.aarch64 and NetworkManager-1:1.36.0-7.el8_6.aarch64
  - cannot install the best update candidate for package NetworkManager-cloud-setup-1:1.36.0-4.el8.aarch64
  - cannot install the best update candidate for package NetworkManager-1:1.36.0-4.el8.aarch64
 Problem 2: problem with installed package NetworkManager-cloud-setup-1:1.36.0-4.el8.aarch64
  - package NetworkManager-cloud-setup-1:1.36.0-4.el8.aarch64 requires NetworkManager-libnm(aarch-64) = 1:1.36.0-4.el8, but none of the providers can be installed
  - package NetworkManager-cloud-setup-1:1.36.0-7.el8_6.aarch64 requires NetworkManager-libnm(aarch-64) = 1:1.36.0-7.el8_6, but none of the providers can be installed
  - cannot install both NetworkManager-libnm-1:1.36.0-9.el8_6.aarch64 and NetworkManager-libnm-1:1.36.0-4.el8.aarch64
  - cannot install both NetworkManager-libnm-1:1.36.0-9.el8_6.aarch64 and NetworkManager-libnm-1:1.36.0-7.el8_6.aarch64
  - cannot install the best update candidate for package NetworkManager-libnm-1:1.36.0-4.el8.aarch64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@ec2-user]#

Comment 6 peninm 2022-10-30 02:29:53 UTC
Just tried it and still getting the same issue in us-west-2

Comment 7 Radek Bíba 2022-10-30 07:41:41 UTC
Right. The CDN repodata issue with RHEL 8 for ARM64 AppStream has been worked around in us-east-1, so VMs in the (AFAICT) most exposed region can continue to receive updates. Rather than working around the issue in all regions on the RHUI level, we aim to resolve the upstream issue ASAP.

Comment 8 Radek Bíba 2022-10-31 16:00:13 UTC
OK, the repo is now up to date in all regions. Please try one more time.

Comment 9 peninm 2022-10-31 16:24:37 UTC
Just tested it in several different regions and it was working. I'll keep checking the other regions, but looks like the update worked.