Bug 1510582 - CDN generate incorrect EngID pem files on pegas host
Summary: CDN generate incorrect EngID pem files on pegas host
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: releng
Version: 7.4
Hardware: ppc64le
OS: Linux
high
urgent
Target Milestone: rc
: ---
Assignee: Jon Disnard
QA Contact: Release Test Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-07 17:35 UTC by Qian Cai
Modified: 2018-12-07 22:36 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-07 22:36:55 UTC
Target Upstream Version:


Attachments (Terms of Use)
Close of SVCRH00284 (169.81 KB, image/png)
2017-11-08 02:38 UTC, John Sefler
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1526622 None CLOSED the productid plugin should never delete a /etc/pki/product-default/<ID>.pem cert provided by the redhat-release-<VARIAN... 2019-10-15 12:00:39 UTC

Internal Links: 1526622

Description Qian Cai 2017-11-07 17:35:31 UTC
Description of problem:
Every time run yum to install from rhel-7-for-power-9-extras-rpms repo, it generated 279.pem and 362.pem under /etc/pki/product which caused issues later on .

Version-Release number of selected component (if applicable):
subscription-manager-1.19.23-1.el7_4.ppc64le
RHEL-ALT-7.4-20171030.0

How reproducible:
always

Comment 1 Qian Cai 2017-11-07 18:07:38 UTC
Here is more output for debug. This is tested in CDN stage.

# subscription-manager attach --auto


Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for Power 9
Status:       Subscribed

# subscription-manager list

Product Name:   Red Hat Enterprise Linux for Power 9
Product ID:     420
Version:        7.4
Arch:           ppc64le
Status:         Subscribed
Status Details: 
Starts:         11/01/2017
Ends:           10/31/2018

# ls -l /etc/pki/product
<nothing>

# ls -l /etc/pki/product-default/
total 4
-rw-r--r--. 1 root root 2183 Oct 26 10:35 420.pem

# yum install runc
Loaded plugins: product-id, search-disabled-repos, subscription-manager
rhel-7-for-power-9-extras-beta-rpms                                                                                                                                    | 4.0 kB  00:00:00     
rhel-7-for-power-9-extras-rpms                                                                                                                                         | 4.0 kB  00:00:00     
rhel-7-for-power-9-rpms                                                                                                                                                | 4.0 kB  00:00:00     
(1/9): rhel-7-for-power-9-extras-beta-rpms/ppc64le/group                                                                                                               |  104 B  00:00:00     
(2/9): rhel-7-for-power-9-extras-beta-rpms/ppc64le/updateinfo                                                                                                          |   76 B  00:00:00     
(3/9): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/group                                                                                                            |  104 B  00:00:00     
(4/9): rhel-7-for-power-9-extras-beta-rpms/ppc64le/primary_db                                                                                                          |  43 kB  00:00:00     
(5/9): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/updateinfo                                                                                                       |  27 kB  00:00:00     
(6/9): rhel-7-for-power-9-extras-rpms/7Server/ppc64le/primary_db                                                                                                       |  76 kB  00:00:00     
(7/9): rhel-7-for-power-9-rpms/7Server/ppc64le/updateinfo                                                                                                              |  26 kB  00:00:00     
(8/9): rhel-7-for-power-9-rpms/7Server/ppc64le/group                                                                                                                   | 666 kB  00:00:01     
(9/9): rhel-7-for-power-9-rpms/7Server/ppc64le/primary_db                                                                                                              | 4.5 MB  00:00:03     
Resolving Dependencies
--> Running transaction check
---> Package runc.ppc64le 0:1.0.0-14.rc4dev.git84a082b.el7 will be installed
--> Processing Dependency: criu for package: runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le
--> Running transaction check
---> Package criu.ppc64le 0:2.12-4.el7a will be installed
--> Processing Dependency: libprotobuf-c.so.1(LIBPROTOBUF_C_1.0.0)(64bit) for package: criu-2.12-4.el7a.ppc64le
--> Processing Dependency: libnet.so.1()(64bit) for package: criu-2.12-4.el7a.ppc64le
--> Processing Dependency: libprotobuf-c.so.1()(64bit) for package: criu-2.12-4.el7a.ppc64le
--> Running transaction check
---> Package libnet.ppc64le 0:1.1.6-7.el7 will be installed
---> Package protobuf-c.ppc64le 0:1.0.2-3.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================
 Package                             Arch                             Version                                                   Repository                                               Size
==============================================================================================================================================================================================
Installing:
 runc                                ppc64le                          1.0.0-14.rc4dev.git84a082b.el7                            rhel-7-for-power-9-extras-rpms                          2.2 M
Installing for dependencies:
 criu                                ppc64le                          2.12-4.el7a                                               beaker-Server                                           390 k
 libnet                              ppc64le                          1.1.6-7.el7                                               beaker-Server                                            61 k
 protobuf-c                          ppc64le                          1.0.2-3.el7                                               beaker-Server                                            29 k

Transaction Summary
==============================================================================================================================================================================================
Install  1 Package (+3 Dependent packages)

Total download size: 2.7 M
Installed size: 12 M
Is this ok [y/d/N]: y
Downloading packages:
(1/4): libnet-1.1.6-7.el7.ppc64le.rpm                                                                                                                                  |  61 kB  00:00:00     
(2/4): protobuf-c-1.0.2-3.el7.ppc64le.rpm                                                                                                                              |  29 kB  00:00:00     
(3/4): criu-2.12-4.el7a.ppc64le.rpm                                                                                                                                    | 390 kB  00:00:00     
warning: /var/cache/yum/ppc64le/7Server/rhel-7-for-power-9-extras-rpms/packages/runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY00 ETA 
Public key for runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le.rpm is not installed
(4/4): runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le.rpm                                                                                                                 | 2.2 MB  00:00:02     
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                         1.1 MB/s | 2.7 MB  00:00:02     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0xFD431D51:
 Userid     : "Red Hat, Inc. (release key 2) <security@redhat.com>"
 Fingerprint: 567e 347a d004 4ade 55ba 8a5f 199e 2f91 fd43 1d51
 Package    : redhat-release-server-7.4-23.el7a.ppc64le (@beaker-Server/7.4)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Importing GPG key 0x2FA658E0:
 Userid     : "Red Hat, Inc. (auxiliary key) <security@redhat.com>"
 Fingerprint: 43a6 e49c 4a38 f4be 9abf 2a53 4568 9c88 2fa6 58e0
 Package    : redhat-release-server-7.4-23.el7a.ppc64le (@beaker-Server/7.4)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libnet-1.1.6-7.el7.ppc64le                                                                                                                                                 1/4 
  Installing : protobuf-c-1.0.2-3.el7.ppc64le                                                                                                                                             2/4 
  Installing : criu-2.12-4.el7a.ppc64le                                                                                                                                                   3/4 
  Installing : runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le                                                                                                                                4/4 
rhel-7-for-power-9-extras-beta-rpms/ppc64le/productid                                                                                                                  | 2.1 kB  00:00:00     
rhel-7-for-power-9-extras-rpms/7Server/ppc64le/productid                                                                                                               | 2.1 kB  00:00:00     
rhel-7-for-power-9-rpms/7Server/ppc64le/productid                                                                                                                      | 2.1 kB  00:00:00     
  Verifying  : protobuf-c-1.0.2-3.el7.ppc64le                                                                                                                                             1/4 
  Verifying  : criu-2.12-4.el7a.ppc64le                                                                                                                                                   2/4 
  Verifying  : libnet-1.1.6-7.el7.ppc64le                                                                                                                                                 3/4 
  Verifying  : runc-1.0.0-14.rc4dev.git84a082b.el7.ppc64le                                                                                                                                4/4 

Installed:
  runc.ppc64le 0:1.0.0-14.rc4dev.git84a082b.el7                                                                                                                                               

Dependency Installed:
  criu.ppc64le 0:2.12-4.el7a                                  libnet.ppc64le 0:1.1.6-7.el7                                  protobuf-c.ppc64le 0:1.0.2-3.el7                                 

Complete!

# ls -l /etc/pki/product
total 8
-rw-r--r--. 1 root root 2199 Nov  7 12:59 279.pem
-rw-r--r--. 1 root root 2195 Nov  7 12:59 362.pem

# subscription-manager list

+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux for Power, little endian
Product ID:     279
Version:        7.4
Arch:           ppc64le
Status:         Subscribed
Status Details: 
Starts:         11/01/2017
Ends:           10/31/2018

Product Name:   Red Hat Enterprise Linux for Power 9
Product ID:     362
Version:        7.4 Beta
Arch:           ppc64le
Status:         Subscribed
Status Details: 
Starts:         11/01/2017
Ends:           10/31/2018

Product Name:   Red Hat Enterprise Linux for Power 9
Product ID:     420
Version:        7.4
Arch:           ppc64le
Status:         Subscribed
Status Details: 
Starts:         11/01/2017
Ends:           10/31/2018

Comment 6 John Sefler 2017-11-08 02:38:48 UTC
Created attachment 1349191 [details]
Close of SVCRH00284

Please see attached screenshot from product admin tool.

The following beta content set repos have the wrong Meta Data tag "rhel-alt-7-power9".  This is the required tag for GA engid 420 which explains why the beta repo appears after attaching SKU RH00284 on a system with 420 installed.  Once the yum transaction from comment 0 installs the first rpm from  the beta repo, product cert 362 gets subsequently installed via the product-id yum plugin.

  rhel-7-for-power-9-extras-beta-rpms
  rhel-7-for-power-9-extras-beta-debug-rpms
  rhel-7-for-power-9-extras-beta-source-rpms


Solution: RCM should fix the meta data tags for those ^^^ three content sets to "rhel-alt-7-ibm-power-9"

Comment 7 Djordje Todorovic 2017-11-08 10:49:45 UTC
fixing metadata in stage for beta repos:

[dtodorov@rcm-dev:cdn][master]$ product-proxy-push-content --eng-server stage content-stage.csv 
# 6499 Metadata rhel-alt-7-power9 -> rhel-alt-7-ibm-power-9
# 6500 Metadata rhel-alt-7-power9 -> rhel-alt-7-ibm-power-9
# 6501 Metadata rhel-alt-7-power9 -> rhel-alt-7-ibm-power-9
engproduct-cli add-eng-content --server stage --content /tmp/push_content.pvBaiw
[dtodorov@rcm-dev:cdn][master]$ engproduct-cli add-eng-content --server stage --content /tmp/push_content.pvBaiw
update rhel-7-for-power-9-extras-beta-rpms

update rhel-7-for-power-9-extras-beta-debug-rpms

update rhel-7-for-power-9-extras-beta-source-rpms

Comment 8 Qian Cai 2017-11-08 15:06:49 UTC
This seems still broken. Any installation by yum from enabled rhel-7-for-power-9-extras-rpms and rhel-7-for-power-9-rpms will drop 279 and 362.pem and DELETE 420.pem from /etc/pki/product-default .

# openssl x509 -in /var/cache/yum/ppc64le/7Server/rhel-7-for-power-9-extras-rpms/productid -text -noout
...
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            1.3.6.1.4.1.2312.9.1.279.1: 
                .1Red Hat Enterprise Linux for Power, little endian
            1.3.6.1.4.1.2312.9.1.279.2: 
                ..7.4
            1.3.6.1.4.1.2312.9.1.279.3: 
                ..ppc64le
            1.3.6.1.4.1.2312.9.1.279.4: 
                ..rhel-7,rhel-7-ibm-power-le
...

# openssl x509 -in /var/cache/yum/ppc64le/7Server/rhel-7-for-power-9-rpms/productid -text -noout
...
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            1.3.6.1.4.1.2312.9.1.362.1: 
                .$Red Hat Enterprise Linux for Power 9
            1.3.6.1.4.1.2312.9.1.362.2: 
                ..7.4 Beta
            1.3.6.1.4.1.2312.9.1.362.3: 
                ..ppc64le
            1.3.6.1.4.1.2312.9.1.362.4: 
                .!rhel-alt-7,rhel-alt-7-ibm-power-9
...

Comment 9 Li Bin Liu 2017-11-08 16:19:11 UTC
Hi Qian,

I might have some different findings/views, here's my comments:

I agree with John's comments in above Comment 3/6 and I have also had a check, I find that the package 'runc' mentioned in this bug only exists in extras beta repo but not in extras GA repo, that is, if the extras beta is disabled, then the pkg 'runc' can not be installed as shown in [1], and if the extras beta is enabled, then the pkg 'runc' can be installed with 362.pem downloaded, but if I just install a pkg from extras GA repo like 'python-itsdangerous' with extras beta repo disabled, there is no any redundant pem files downloaded into system. Therefore, I think the key issue here should be that the extras beta repo should not be appearing since the system just installed the GA product 420, for this issue, please see my comments in the bug https://bugzilla.redhat.com/show_bug.cgi?id=1509877#c14.

[1]
[root@ibm-p8-kvm-06-guest-03 ~]# repoquery --show-dupes --all --repoid=rhel-7-for-power-9-extras-rpms --qf "%{name}-%{version}-%{release}.%{arch}" |sort -u
cockpit-dashboard-151-1.el7.ppc64le
cockpit-docker-151-1.el7.ppc64le
cockpit-packagekit-151-1.el7.ppc64le
cockpit-pcp-151-1.el7.ppc64le
cockpit-storaged-151-1.el7.noarch
python-itsdangerous-0.23-2.el7.noarch

[root@ibm-p8-kvm-06-guest-03 ~]# repoquery --show-dupes --all --repoid=rhel-7-for-power-9-extras-beta-rpms --qf "%{name}-%{version}-%{release}.%{arch}" |sort -u|grep runc
runc-1.0.0-12.1.gitf8ce01d.el7.ppc64le

[root@ibm-p8-kvm-06-guest-03 ~]# subscription-manager repos --disable=rhel-7-for-power-9-extras-beta-rpms
Repository 'rhel-7-for-power-9-extras-beta-rpms' is disabled for this system.

[root@ibm-p8-kvm-06-guest-03 ~]# yum install runc
Loaded plugins: product-id, search-disabled-repos, subscription-manager
rhel-7-for-power-9-extras-rpms                                                                                                                                                          | 3.8 kB  00:00:00     
rhel-7-for-power-9-rpms                                                                                                                                                                 | 4.0 kB  00:00:00     
No package runc available.
Error: Nothing to do


My system info is listed below, you can also have a try in it:
$ ssh root@ibm-p8-kvm-06-guest-03.rhts.eng.bos.redhat.com (user/pwd: root/redhat)

Best Regards,
Libin

Comment 10 Qian Cai 2017-11-08 16:35:57 UTC
Let's just use this BZ focus on comment #8.

I'll try to debug your power9-extra repo inaccessible issue in bz 1509877 where I did not see in this baremetal machine (not kvm like yours).

Comment 13 Qian Cai 2017-11-09 18:25:55 UTC
No longer an issue in production.

Comment 19 Qian Cai 2018-01-03 19:23:06 UTC
OK, I found a workaround for the above machine shortage problem.

This seems still some problem on stage. It looks like the beta repo for RHEL 7.5-ALT beta is not setup properly. After attached to a beta power9 subscription, a GA repo will be enabled by default. As the results, any installation from that GA repo will result in the beta pem (362.pem) being removed and the GA pem (420.pem) being installed.

# subscription-manager attach --auto
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for Power 9 Beta

Status:       Subscribed
# yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
rhel-7-for-power-9-rpms                                 | 4.0 kB     00:00     
(1/3): rhel-7-for-power-9-rpms/7Server/ppc64le/updateinfo |  87 kB   00:00     
(2/3): rhel-7-for-power-9-rpms/7Server/ppc64le/group      | 666 kB   00:01     
(3/3): rhel-7-for-power-9-rpms/7Server/ppc64le/primary_db | 6.5 MB   00:04

# subscription-manager list
+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux for Power 9 Beta
Product ID:     362
Version:        7.5 Beta
Arch:           ppc64le
Status:         Subscribed
Status Details: 
Starts:         11/01/2017
Ends:           01/10/2018

Comment 20 Qian Cai 2018-01-16 21:49:47 UTC
Reopen this as this is still an issue in the CDN stage for RHEL 7.5 beta.

# openssl x509 -in /var/cache/yum/ppc64le/7Server/rhel-7-for-power-9-rpms/productid -text -noout
...
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            1.3.6.1.4.1.2312.9.1.420.1: 
                .$Red Hat Enterprise Linux for Power 9
            1.3.6.1.4.1.2312.9.1.420.2: 
                ..7.4
            1.3.6.1.4.1.2312.9.1.420.3: 
                ..ppc64le
            1.3.6.1.4.1.2312.9.1.420.4: 
                ..rhel-alt-7,rhel-alt-7-power9
...

As the result the GA pem (420) will be pulled down from the server.

Comment 25 Qian Cai 2018-01-24 17:36:37 UTC
This is now happening in 7.5 beta production as well.

# yum install docker
...
rhel-7-for-power-9-beta-rpms/ppc64le/productid           | 2.1 kB     00:00     
rhel-7-for-power-9-rpms/7Server/ppc64le/productid        | 2.1 kB     00:00     
...

# ls -l /etc/pki/product
total 8
-rw-r--r--. 1 root root 2195 Jan 24 12:28 362.pem
-rw-r--r--. 1 root root 2183 Jan 24 12:28 420.pem

# ls -l /etc/pki/product-default/
total 0

It absolutely need to enable to GA channel - rhel-7-for-power-9-rpms because it has some packages like PyYAML (in order to install docker) that never available in rhel-7-for-power-9-beta-rpms.

Comment 33 Lubos Kocman 2018-05-28 15:00:25 UTC
I believe this is all set. Especially now when we've unified engid and both Beta and GA have one. Moving to RTT to confirm that we're all set here.

Comment 34 Liu Song 2018-07-03 03:42:23 UTC
I am trying to verify the bug on RHEL-ALT-7.6-DevelPhaseExit-1.0 for CDN QA, but there is no extra directory for beta.

See the link:
http://cdn.qa.redhat.com/content/beta/rhel-alt/server/7/7Server/power9/ppc64le/

I guess the bug does not exist now, because of the engid is unified for Beta and GA.

Comment 36 Jon Disnard 2018-12-07 22:36:55 UTC
movign to close because it works now.


Note You need to log in before you can comment on or make changes to this bug.