Bug 1510582
| Summary: | CDN generate incorrect EngID pem files on pegas host | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Qian Cai <qcai> | ||||
| Component: | releng | Assignee: | Jon Disnard <jdisnard> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Release Test Team <release-test-team-automation> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.4 | CC: | dgilmore, dtodorov, jreznik, jsefler, khowell, lfriedma, liliu, lkocman, qcai, sgraf, soliu | ||||
| Target Milestone: | rc | Keywords: | Reopened | ||||
| Target Release: | --- | ||||||
| Hardware: | ppc64le | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-12-07 22:36:55 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Qian Cai
2017-11-07 17:35:31 UTC
Here is more output for debug. This is tested in CDN stage.
# subscription-manager attach --auto
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for Power 9
Status: Subscribed
# subscription-manager list
Product Name: Red Hat Enterprise Linux for Power 9
Product ID: 420
Version: 7.4
Arch: ppc64le
Status: Subscribed
Status Details:
Starts: 11/01/2017
Ends: 10/31/2018
# ls -l /etc/pki/product
<nothing>
# ls -l /etc/pki/product-default/
total 4
-rw-r--r--. 1 root root 2183 Oct 26 10:35 420.pem
# yum install runc
Loaded plugins: product-id, search-disabled-repos, subscription-manager
rhel-7-for-power-9-extras-beta-rpms | 4.0 kB 00:00:00
rhel-7-for-power-9-extras-rpms | 4.0 kB 00:00:00
rhel-7-for-power-9-rpms | 4.0 kB 00:00:00
(1/9): rhel-7-for-power-9-extras-beta-rpms/ppc64le/group | 104 B 00:00:00
(2/9): rhel-7-for-power-9-extras-beta-rpms/ppc64le/updateinfo | 76 B 00:00:00
(3/9): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/group | 104 B 00:00:00
(4/9): rhel-7-for-power-9-extras-beta-rpms/ppc64le/primary_db | 43 kB 00:00:00
(5/9): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/updateinfo | 27 kB 00:00:00
(6/9): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/primary_db | 76 kB 00:00:00
(7/9): rhel-7-for-power-9-rpms/7Server/ppc64le/updateinfo | 26 kB 00:00:00
(8/9): rhel-7-for-power-9-rpms/7Server/ppc64le/group | 666 kB 00:00:01
(9/9): rhel-7-for-power-9-rpms/7Server/ppc64le/primary_db | 4.5 MB 00:00:03
Resolving Dependencies
--> Running transaction check
---> Package runc.ppc64le 0:1.0.0-14.rc4dev.git84a082b.el7 will be installed
--> Processing Dependency: criu for package: runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le
--> Running transaction check
---> Package criu.ppc64le 0:2.12-4.el7a will be installed
--> Processing Dependency: libprotobuf-c.so.1(LIBPROTOBUF_C_1.0.0)(64bit) for package: criu-2.12-4.el7a.ppc64le
--> Processing Dependency: libnet.so.1()(64bit) for package: criu-2.12-4.el7a.ppc64le
--> Processing Dependency: libprotobuf-c.so.1()(64bit) for package: criu-2.12-4.el7a.ppc64le
--> Running transaction check
---> Package libnet.ppc64le 0:1.1.6-7.el7 will be installed
---> Package protobuf-c.ppc64le 0:1.0.2-3.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==============================================================================================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================================================================================
Installing:
runc ppc64le 1.0.0-14.rc4dev.git84a082b.el7 rhel-7-for-power-9-extras-rpms 2.2 M
Installing for dependencies:
criu ppc64le 2.12-4.el7a beaker-Server 390 k
libnet ppc64le 1.1.6-7.el7 beaker-Server 61 k
protobuf-c ppc64le 1.0.2-3.el7 beaker-Server 29 k
Transaction Summary
==============================================================================================================================================================================================
Install 1 Package (+3 Dependent packages)
Total download size: 2.7 M
Installed size: 12 M
Is this ok [y/d/N]: y
Downloading packages:
(1/4): libnet-1.1.6-7.el7.ppc64le.rpm | 61 kB 00:00:00
(2/4): protobuf-c-1.0.2-3.el7.ppc64le.rpm | 29 kB 00:00:00
(3/4): criu-2.12-4.el7a.ppc64le.rpm | 390 kB 00:00:00
warning: /var/cache/yum/ppc64le/7Server/rhel-7-for-power-9-extras-rpms/packages/runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY00 ETA
Public key for runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le.rpm is not installed
(4/4): runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le.rpm | 2.2 MB 00:00:02
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.1 MB/s | 2.7 MB 00:00:02
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0xFD431D51:
Userid : "Red Hat, Inc. (release key 2) <security>"
Fingerprint: 567e 347a d004 4ade 55ba 8a5f 199e 2f91 fd43 1d51
Package : redhat-release-server-7.4-23.el7a.ppc64le (@beaker-Server/7.4)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Importing GPG key 0x2FA658E0:
Userid : "Red Hat, Inc. (auxiliary key) <security>"
Fingerprint: 43a6 e49c 4a38 f4be 9abf 2a53 4568 9c88 2fa6 58e0
Package : redhat-release-server-7.4-23.el7a.ppc64le (@beaker-Server/7.4)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : libnet-1.1.6-7.el7.ppc64le 1/4
Installing : protobuf-c-1.0.2-3.el7.ppc64le 2/4
Installing : criu-2.12-4.el7a.ppc64le 3/4
Installing : runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le 4/4
rhel-7-for-power-9-extras-beta-rpms/ppc64le/productid | 2.1 kB 00:00:00
rhel-7-for-power-9-extras-rpms/7Server/ppc64le/productid | 2.1 kB 00:00:00
rhel-7-for-power-9-rpms/7Server/ppc64le/productid | 2.1 kB 00:00:00
Verifying : protobuf-c-1.0.2-3.el7.ppc64le 1/4
Verifying : criu-2.12-4.el7a.ppc64le 2/4
Verifying : libnet-1.1.6-7.el7.ppc64le 3/4
Verifying : runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le 4/4
Installed:
runc.ppc64le 0:1.0.0-14.rc4dev.git84a082b.el7
Dependency Installed:
criu.ppc64le 0:2.12-4.el7a libnet.ppc64le 0:1.1.6-7.el7 protobuf-c.ppc64le 0:1.0.2-3.el7
Complete!
# ls -l /etc/pki/product
total 8
-rw-r--r--. 1 root root 2199 Nov 7 12:59 279.pem
-rw-r--r--. 1 root root 2195 Nov 7 12:59 362.pem
# subscription-manager list
+-------------------------------------------+
Installed Product Status
+-------------------------------------------+
Product Name: Red Hat Enterprise Linux for Power, little endian
Product ID: 279
Version: 7.4
Arch: ppc64le
Status: Subscribed
Status Details:
Starts: 11/01/2017
Ends: 10/31/2018
Product Name: Red Hat Enterprise Linux for Power 9
Product ID: 362
Version: 7.4 Beta
Arch: ppc64le
Status: Subscribed
Status Details:
Starts: 11/01/2017
Ends: 10/31/2018
Product Name: Red Hat Enterprise Linux for Power 9
Product ID: 420
Version: 7.4
Arch: ppc64le
Status: Subscribed
Status Details:
Starts: 11/01/2017
Ends: 10/31/2018
Created attachment 1349191 [details] Close of SVCRH00284 Please see attached screenshot from product admin tool. The following beta content set repos have the wrong Meta Data tag "rhel-alt-7-power9". This is the required tag for GA engid 420 which explains why the beta repo appears after attaching SKU RH00284 on a system with 420 installed. Once the yum transaction from comment 0 installs the first rpm from the beta repo, product cert 362 gets subsequently installed via the product-id yum plugin. rhel-7-for-power-9-extras-beta-rpms rhel-7-for-power-9-extras-beta-debug-rpms rhel-7-for-power-9-extras-beta-source-rpms Solution: RCM should fix the meta data tags for those ^^^ three content sets to "rhel-alt-7-ibm-power-9" fixing metadata in stage for beta repos: [dtodorov@rcm-dev:cdn][master]$ product-proxy-push-content --eng-server stage content-stage.csv # 6499 Metadata rhel-alt-7-power9 -> rhel-alt-7-ibm-power-9 # 6500 Metadata rhel-alt-7-power9 -> rhel-alt-7-ibm-power-9 # 6501 Metadata rhel-alt-7-power9 -> rhel-alt-7-ibm-power-9 engproduct-cli add-eng-content --server stage --content /tmp/push_content.pvBaiw [dtodorov@rcm-dev:cdn][master]$ engproduct-cli add-eng-content --server stage --content /tmp/push_content.pvBaiw update rhel-7-for-power-9-extras-beta-rpms update rhel-7-for-power-9-extras-beta-debug-rpms update rhel-7-for-power-9-extras-beta-source-rpms This seems still broken. Any installation by yum from enabled rhel-7-for-power-9-extras-rpms and rhel-7-for-power-9-rpms will drop 279 and 362.pem and DELETE 420.pem from /etc/pki/product-default .
# openssl x509 -in /var/cache/yum/ppc64le/7Server/rhel-7-for-power-9-extras-rpms/productid -text -noout
...
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
1.3.6.1.4.1.2312.9.1.279.1:
.1Red Hat Enterprise Linux for Power, little endian
1.3.6.1.4.1.2312.9.1.279.2:
..7.4
1.3.6.1.4.1.2312.9.1.279.3:
..ppc64le
1.3.6.1.4.1.2312.9.1.279.4:
..rhel-7,rhel-7-ibm-power-le
...
# openssl x509 -in /var/cache/yum/ppc64le/7Server/rhel-7-for-power-9-rpms/productid -text -noout
...
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
1.3.6.1.4.1.2312.9.1.362.1:
.$Red Hat Enterprise Linux for Power 9
1.3.6.1.4.1.2312.9.1.362.2:
..7.4 Beta
1.3.6.1.4.1.2312.9.1.362.3:
..ppc64le
1.3.6.1.4.1.2312.9.1.362.4:
.!rhel-alt-7,rhel-alt-7-ibm-power-9
...
Hi Qian, I might have some different findings/views, here's my comments: I agree with John's comments in above Comment 3/6 and I have also had a check, I find that the package 'runc' mentioned in this bug only exists in extras beta repo but not in extras GA repo, that is, if the extras beta is disabled, then the pkg 'runc' can not be installed as shown in [1], and if the extras beta is enabled, then the pkg 'runc' can be installed with 362.pem downloaded, but if I just install a pkg from extras GA repo like 'python-itsdangerous' with extras beta repo disabled, there is no any redundant pem files downloaded into system. Therefore, I think the key issue here should be that the extras beta repo should not be appearing since the system just installed the GA product 420, for this issue, please see my comments in the bug https://bugzilla.redhat.com/show_bug.cgi?id=1509877#c14. [1] [root@ibm-p8-kvm-06-guest-03 ~]# repoquery --show-dupes --all --repoid=rhel-7-for-power-9-extras-rpms --qf "%{name}-%{version}-%{release}.%{arch}" |sort -u cockpit-dashboard-151-1.el7.ppc64le cockpit-docker-151-1.el7.ppc64le cockpit-packagekit-151-1.el7.ppc64le cockpit-pcp-151-1.el7.ppc64le cockpit-storaged-151-1.el7.noarch python-itsdangerous-0.23-2.el7.noarch [root@ibm-p8-kvm-06-guest-03 ~]# repoquery --show-dupes --all --repoid=rhel-7-for-power-9-extras-beta-rpms --qf "%{name}-%{version}-%{release}.%{arch}" |sort -u|grep runc runc-1.0.0-12.1.gitf8ce01d.el7.ppc64le [root@ibm-p8-kvm-06-guest-03 ~]# subscription-manager repos --disable=rhel-7-for-power-9-extras-beta-rpms Repository 'rhel-7-for-power-9-extras-beta-rpms' is disabled for this system. [root@ibm-p8-kvm-06-guest-03 ~]# yum install runc Loaded plugins: product-id, search-disabled-repos, subscription-manager rhel-7-for-power-9-extras-rpms | 3.8 kB 00:00:00 rhel-7-for-power-9-rpms | 4.0 kB 00:00:00 No package runc available. Error: Nothing to do My system info is listed below, you can also have a try in it: $ ssh root.eng.bos.redhat.com (user/pwd: root/redhat) Best Regards, Libin Let's just use this BZ focus on comment #8. I'll try to debug your power9-extra repo inaccessible issue in bz 1509877 where I did not see in this baremetal machine (not kvm like yours). No longer an issue in production. OK, I found a workaround for the above machine shortage problem.
This seems still some problem on stage. It looks like the beta repo for RHEL 7.5-ALT beta is not setup properly. After attached to a beta power9 subscription, a GA repo will be enabled by default. As the results, any installation from that GA repo will result in the beta pem (362.pem) being removed and the GA pem (420.pem) being installed.
# subscription-manager attach --auto
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for Power 9 Beta
Status: Subscribed
# yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
rhel-7-for-power-9-rpms | 4.0 kB 00:00
(1/3): rhel-7-for-power-9-rpms/7Server/ppc64le/updateinfo | 87 kB 00:00
(2/3): rhel-7-for-power-9-rpms/7Server/ppc64le/group | 666 kB 00:01
(3/3): rhel-7-for-power-9-rpms/7Server/ppc64le/primary_db | 6.5 MB 00:04
# subscription-manager list
+-------------------------------------------+
Installed Product Status
+-------------------------------------------+
Product Name: Red Hat Enterprise Linux for Power 9 Beta
Product ID: 362
Version: 7.5 Beta
Arch: ppc64le
Status: Subscribed
Status Details:
Starts: 11/01/2017
Ends: 01/10/2018
Reopen this as this is still an issue in the CDN stage for RHEL 7.5 beta.
# openssl x509 -in /var/cache/yum/ppc64le/7Server/rhel-7-for-power-9-rpms/productid -text -noout
...
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
1.3.6.1.4.1.2312.9.1.420.1:
.$Red Hat Enterprise Linux for Power 9
1.3.6.1.4.1.2312.9.1.420.2:
..7.4
1.3.6.1.4.1.2312.9.1.420.3:
..ppc64le
1.3.6.1.4.1.2312.9.1.420.4:
..rhel-alt-7,rhel-alt-7-power9
...
As the result the GA pem (420) will be pulled down from the server.
This is now happening in 7.5 beta production as well. # yum install docker ... rhel-7-for-power-9-beta-rpms/ppc64le/productid | 2.1 kB 00:00 rhel-7-for-power-9-rpms/7Server/ppc64le/productid | 2.1 kB 00:00 ... # ls -l /etc/pki/product total 8 -rw-r--r--. 1 root root 2195 Jan 24 12:28 362.pem -rw-r--r--. 1 root root 2183 Jan 24 12:28 420.pem # ls -l /etc/pki/product-default/ total 0 It absolutely need to enable to GA channel - rhel-7-for-power-9-rpms because it has some packages like PyYAML (in order to install docker) that never available in rhel-7-for-power-9-beta-rpms. I believe this is all set. Especially now when we've unified engid and both Beta and GA have one. Moving to RTT to confirm that we're all set here. I am trying to verify the bug on RHEL-ALT-7.6-DevelPhaseExit-1.0 for CDN QA, but there is no extra directory for beta. See the link: http://cdn.qa.redhat.com/content/beta/rhel-alt/server/7/7Server/power9/ppc64le/ I guess the bug does not exist now, because of the engid is unified for Beta and GA. movign to close because it works now. |