Bug 1288602

Summary: RHEL KVM Guest image - product cert for both Beta and RHEL in /etc/pki/product[-default]
Product: Red Hat Enterprise Linux 7 Reporter: Harald Jensås <hjensas>
Component: rhel-guest-imageAssignee: Lubos Kocman <lkocman>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: high    
Version: 7.2CC: creynold, fdeutsch, hjensas, jgreguske, jswensso, linl, lkocman, mburgerh, mkalyat, rbarry, wshi
Target Milestone: rcKeywords: ZStream
Target Release: 7.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: node
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1311958 (view as bug list) Environment:
Last Closed: 2016-11-29 11:29:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1301891, 1311958    

Description Harald Jensås 2015-12-04 17:17:16 UTC
Description of problem:

There are two, different, 69.pem certificates installed. I belive the /etc/pki/product/69.pem is the Beta product certificate. This causes subcription-manager to report "Overall Status: Invalid".

Running "subscription-manager list" we can see that both Beta and RHEL Server product is installed.

[root@host ~]# subscription-manager list

+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux 7 Public Beta
Product ID:     226
Version:        7.0 Beta
Arch:           x86_64
Status:         Not Subscribed
Status Details: Not supported by a valid subscription.
Starts:         
Ends:           

Product Name:   Red Hat Enterprise Linux Server
Product ID:     69
Version:        7.2
Arch:           x86_64
Status:         Subscribed
Status Details: 
Starts:         04/24/2013
Ends:           12/31/2021


Looking at product certs, there are two different certs.

[root@host ~]# cat /etc/pki/product/69.pem 
-----BEGIN CERTIFICATE-----
MIIGGjCCBAKgAwIBAgIJALDxRLt/tU9JMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYD
VQQGEwJVUzEXMBUGA1UECAwOTm9ydGggQ2Fyb2xpbmExFjAUBgNVBAoMDVJlZCBI
YXQsIEluYy4xGDAWBgNVBAsMD1JlZCBIYXQgTmV0d29yazEuMCwGA1UEAwwlUmVk
IEhhdCBFbnRpdGxlbWVudCBQcm9kdWN0IEF1dGhvcml0eTEkMCIGCSqGSIb3DQEJ
ARYVY2Etc3VwcG9ydEByZWRoYXQuY29tMB4XDTEzMTEyMDA5MzgzMFoXDTMzMTEx
NTA5MzgzMFowRDFCMEAGA1UEAww5UmVkIEhhdCBQcm9kdWN0IElEIFs5YjNhMDQy
Mi0zYjYxLTQ0OWUtYWY0My04NmMyNzQzMjZkYjBdMIICIjANBgkqhkiG9w0BAQEF
AAOCAg8AMIICCgKCAgEAxj9J04z+Ezdyx1U33kFftLv0ntNS1BSeuhoZLDhs18yk
sepG7hXXtHh2CMFfLZmTjAyL9i1XsxykQpVQdXTGpUF33C2qBQHB5glYs9+d781x
8p8m8zFxbPcW82TIJXbgW3ErVh8vk5qCbG1cCAAHb+DWMq0EAyy1bl/JgAghYNGB
RvKJObTdCrdpYh02KUqBLkSPZHvo6DUJFN37MXDpVeQq9VtqRjpKLLwuEfXb0Y7I
5xEOrR3kYbOaBAWVt3mYZ1t0L/KfY2jVOdU5WFyyB9PhbMdLi1xE801j+GJrwcLa
xmqvj4UaICRzcPATP86zVM1BBQa+lilkRQes5HyjZzZDiGYudnXhbqmLo/n0cuXo
QBVVjhzRTMx71Eiiahmiw+U1vGqkHhQNxb13HtN1lcAhUCDrxxeMvrAjYdWpYlpI
yW3NssPWt1YUHidMBSAJ4KctIf91dyE93aStlxwC/QnyFsZOmcEsBzVCnz9GmWMl
1/6XzBS1yDUqByklx0TLH+z/sK9A+O2rZAy1mByCYwVxvbOZhnqGxAuToIS+A81v
5hCjsCiOScVB+cil30YBu0cH85RZ0ILNkHdKdrLLWW4wjphK2nBn2g2i3+ztf+nQ
ED2pQqZ/rhuW79jcyCZl9kXqe1wOdF0Cwah4N6/3LzIXEEKyEJxNqQwtNc2IVE8C
AwEAAaOBozCBoDAJBgNVHRMEAjAAMDgGDCsGAQQBkggJAYFiAQQoDCZSZWQgSGF0
IEVudGVycHJpc2UgTGludXggNyBQdWJsaWMgQmV0YTAaBgwrBgEEAZIICQGBYgIE
CgwINy4wIEJldGEwGAYMKwYBBAGSCAkBgWIDBAgMBng4Nl82NDAjBgwrBgEEAZII
CQGBYgQEEwwRcmhlbC03LWV2ZXJ5dGhpbmcwDQYJKoZIhvcNAQEFBQADggIBAAAd
y1SobfUaiEy4/GS2bR6uurGzKM8IDsmMo3XdsEoMSYey0ufhAcCH5cfLcMk5crPX
63FRR+YLbaCenzZtC03ugPm8oELIKx26dzfTMODRI0In2SkQUEoQeFeCtq2fYb7k
FAB5NgC0kR+hR7tmpMqW0W+g+y2HHnETwwSUq7McquJ8TbJUJ+JnUXpNbhLJAOiD
f/QN9A2xe8+OpHCaIomF/2RgE2J5LZSQGGNk+p+Lid4LkoKOWN6S4bPyqcm6yjRZ
X+AlV+ERTIkW2ViDOBCznp1Q7vn+msaGbl+d3+UN9gexgc+e+xYDh7zY02WEpuk2
CQ2TeBoaH7QpOBctv5kbJT6vkqZltn9WlZ4FmGKMMQOAL/Sz2bP+vqoCs3HPwUSk
aH++Zc1ns9Ve0i1LjVih+bfTWsi1BZIlxYgLOxylenM5t7wUYeTN1amnpSxJirZ1
GWVUvTd4xEnWb+0GF6BDsEqtsEOWmXWCCvv1qyQWNoTafspvD90KYXoGU9clMKdq
0Cs5EAdKPp2jeCldW5hp6fQBB1ctOT2z/DOulGXqyOC67E+ELaHCnBRjYw8XXRq+
clJnnJHyd2yTgpUAQFO4UWGyjOYIsGXELfIIbJYq9O8axIEZs50er+gnp10i3eAQ
mmSbckWQOgq2X0mL4lHw7znZUcP5jy1HEyBloZ1b
-----END CERTIFICATE-----

[root@host ~]# cat /etc/pki/product-default/69.pem 
-----BEGIN CERTIFICATE-----
MIIGDTCCA/WgAwIBAgIJALDxRLt/tU6eMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYD
VQQGEwJVUzEXMBUGA1UECAwOTm9ydGggQ2Fyb2xpbmExFjAUBgNVBAoMDVJlZCBI
YXQsIEluYy4xGDAWBgNVBAsMD1JlZCBIYXQgTmV0d29yazEuMCwGA1UEAwwlUmVk
IEhhdCBFbnRpdGxlbWVudCBQcm9kdWN0IEF1dGhvcml0eTEkMCIGCSqGSIb3DQEJ
ARYVY2Etc3VwcG9ydEByZWRoYXQuY29tMB4XDTE1MDkxMTExMDkyMloXDTM1MDkw
NjExMDkyMlowRDFCMEAGA1UEAww5UmVkIEhhdCBQcm9kdWN0IElEIFtlYjg5ZGVl
Ny1mNzg3LTQ3MjQtYmU3ZC03ZWEzMzJkMGQ5ZmFdMIICIjANBgkqhkiG9w0BAQEF
AAOCAg8AMIICCgKCAgEAxj9J04z+Ezdyx1U33kFftLv0ntNS1BSeuhoZLDhs18yk
sepG7hXXtHh2CMFfLZmTjAyL9i1XsxykQpVQdXTGpUF33C2qBQHB5glYs9+d781x
8p8m8zFxbPcW82TIJXbgW3ErVh8vk5qCbG1cCAAHb+DWMq0EAyy1bl/JgAghYNGB
RvKJObTdCrdpYh02KUqBLkSPZHvo6DUJFN37MXDpVeQq9VtqRjpKLLwuEfXb0Y7I
5xEOrR3kYbOaBAWVt3mYZ1t0L/KfY2jVOdU5WFyyB9PhbMdLi1xE801j+GJrwcLa
xmqvj4UaICRzcPATP86zVM1BBQa+lilkRQes5HyjZzZDiGYudnXhbqmLo/n0cuXo
QBVVjhzRTMx71Eiiahmiw+U1vGqkHhQNxb13HtN1lcAhUCDrxxeMvrAjYdWpYlpI
yW3NssPWt1YUHidMBSAJ4KctIf91dyE93aStlxwC/QnyFsZOmcEsBzVCnz9GmWMl
1/6XzBS1yDUqByklx0TLH+z/sK9A+O2rZAy1mByCYwVxvbOZhnqGxAuToIS+A81v
5hCjsCiOScVB+cil30YBu0cH85RZ0ILNkHdKdrLLWW4wjphK2nBn2g2i3+ztf+nQ
ED2pQqZ/rhuW79jcyCZl9kXqe1wOdF0Cwah4N6/3LzIXEEKyEJxNqQwtNc2IVE8C
AwEAAaOBljCBkzAJBgNVHRMEAjAAMDAGCysGAQQBkggJAUUBBCEMH1JlZCBIYXQg
RW50ZXJwcmlzZSBMaW51eCBTZXJ2ZXIwFAYLKwYBBAGSCAkBRQIEBQwDNy4yMBcG
CysGAQQBkggJAUUDBAgMBng4Nl82NDAlBgsrBgEEAZIICQFFBAQWDBRyaGVsLTcs
cmhlbC03LXNlcnZlcjANBgkqhkiG9w0BAQUFAAOCAgEAbx9SPI/5iWOEKo+hJ/98
ohRdsEO/4uClzInLpjkFtqHCtWVyv/nLhdiV2Xrvw7O+byGQFaehTIE3pmL+tsuQ
YcY9fMt3IZH+WslfMH4MnG+C9jR8exns3+TxshcnK9dykXuGkcWyHDY9YCyJx8n7
0XcwdrnHENsgx5kzv5FGipxwk0DUqZavv8d54H5fJlRKMfCJP4Qe9V67kB714twe
L0ggMh6Y9u68D5iP1BIF0wOEsqvGQ/Qm7KYY9LKwVkxVS4MK0Ysmj60PlLCWJaw7
HGY2nKkvxSnqXFcr43UDJVJYr9pawHf1Sp+yDToxUb1ok1Wnx3nLW6zjGzauMOUa
i6f/bALCjy67e0biNJ9gfknAUOtMj2/ucfRIlLhDXRPv61YDRrBta8bIs7ZiDHdi
R9BpKbqbZxwt8+dLOE9A9BiuF+pyO/vm6yJGY5tPFonKgzQ6JDg1AoVaZvhRt6hz
aNQ0O39nw0WJWlYQUPF7d1oHEdmyz9zZ3daqOyZ1bMyL6ktCMfPBTnKjg0sBKfso
mJKZW7ECMV54mFVetoHMhXK95J2gdj8KCCBRiMklNJ/CmObDLwVgiJ+aJyFWkhLx
D9gahhTI9drID4fuiO0rRIgSAD/22bGJY/OnBPq3Fs5Cvaw619Eqth7XjJCl4POb
Kp/OjOg7U7FJSsdrpQKT0/M=
-----END CERTIFICATE-----


Removing the cert under /etc/pki/product solves the problem, so this cert must be the Beta product cert.

[root@host ~]# rm /etc/pki/product/69.pem 
rm: remove regular file ‘/etc/pki/product/69.pem’? y
[root@host ~]# subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Current

[root@host ~]# subscription-manager list

+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux Server
Product ID:     69
Version:        7.2
Arch:           x86_64
Status:         Subscribed
Status Details: 
Starts:         04/24/2013
Ends:           01/01/2022


Version-Release number of selected component (if applicable):
rhel-guest-image-7.2-20151102.0.x86_64.qcow2

How reproducible:
Every time.

Steps to Reproduce:
1. Create an instance based on the rhel guest image
2. Register the system with "subscription-manager register"
3. Run "subscription-manager status"

Actual results:
[root@host ~]# subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Invalid

Red Hat Enterprise Linux 7 Public Beta:
- Not supported by a valid subscription.


Expected results:
[root@host ~]# subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Current


Additional info:

Comment 1 John Sefler 2015-12-11 15:54:41 UTC
I'm not familiar with rhel-guest-image-7.2-20151102.0.x86_64.qcow2, but whomever created it must have manually renamed product cert 226.pem into the /etc/pki/product/69.pem that you removed.  An rct cat-cert of the certificate that you have pasted in comment 0 for /etc/pki/product/69.pem is NOT product 69; it is indeed product 226 as you have shown in the subscription-manager list --installed.

Product 226 and 69 are NOT the same products.  Product 226 provided tag rhel-7-everything and gave access to content prior to the GA release of RHEL7.0.  I don't believe product 226 was ever a customer facing product.

Bottom line...  this is not a subscription-manager bug.  It is a poorly created qcow2 image.

Comment 2 Harald Jensås 2015-12-16 03:49:53 UTC
Agreed, I have re-assigning to component 'rhel-guest-image'.

Comment 3 Ryan Barry 2015-12-17 15:24:01 UTC
This hasn't been touched since the initial RHEL7 release.

It's also present in all of the EC2 images (still a bug, but present), and every one of the RHEL7 guest images since 7.0, and it looks like it was originally to address bz#800120. In other words, it's been there for a long time.

I'd prefer not to need to maintain this product key or do anything with it for a variety of reasons. If the cert is now included as part of the images, I'm happy to drop it from the kickstart.

There's not a lot of visibility into product certs, what provides which entitlements, how they're owned, etc. Can I expect that everything will "just work" after removing it from the kickstart? Including beta keys for beta, etc?

Comment 4 Wei Shi 2015-12-21 01:57:13 UTC
I think it will being included by redhat-release-* RPMs, i find this information under "How do product certificates get installed?" section of the document below:
https://mojo.redhat.com/groups/release-engineering/blog/2015/09/01/product-certificates-explained

rhel-guest-image 7.1 image doesn't have this bug, the correct 69.pem has been created before anaconda process the 69.pem file in kickstart %post%, but i don't know which program created the correct 69.pem.

Comment 5 Ryan Barry 2015-12-21 08:38:52 UTC
When did the change to product-default happen?

Comment 6 Wei Shi 2015-12-21 09:16:18 UTC
for rhel6:
  since rhel-guest-image-6.7
for rhel7:
  since rhel-guest-image-7.2

redhat-release-server RPM contains product-default/69.pem, if a certificate locate in "product" directory which has the same "Product ID" with the one in "product-default" directory, the certificate in "product" dir will override the later one.

we also need to fix this for rhel6 images.

Comment 7 Wei Shi 2015-12-21 09:31:17 UTC
Find the bug to include default product certificate in redhat-release:

rhel7 ->
https://bugzilla.redhat.com/show_bug.cgi?id=1080007

rhel6 ->
https://bugzilla.redhat.com/show_bug.cgi?id=1080012

Comment 9 Fabian Deutsch 2016-02-23 16:03:43 UTC
IIUIC according to comment 7 this bug can be fixed by not including the product cert in the ks, but rely on the product cert from the redhat-release rpms.

If you agree Ryan, then a new build can be done with the from the ks removed pems to see if this issue is fixed.

Comment 10 Ryan Barry 2016-02-23 16:14:57 UTC
Yes, I agree, though I'm away for the next couple of days, so this wouldn't be done until early next week.

Comment 17 Lubos Kocman 2016-11-29 11:29:56 UTC
This should happen by default as of now the kvm image gets redhat-release-server ... (based on variant) from compose. And redhat-release-server contains default productid certificates.

There should be always just one certificate (which would be Beta in Beta, HTB in Snapshots and GA in GA). If you get one with e.g. Beta cert you've used beta kvm-image.

Is there something that I'm missing. As of RHEL-7.4 (not entirelt sure about 7.3) the kvm image is being produced as part of compose.

http://download.devel.redhat.com/nightly/latest-RHEL-7/compose/Server/x86_64/images/

Closing as notabug. Feel free to re-open the issue. You guys can sync-up with me and we test image in Alpha, Beta, Snapshot, RC composes ... wheter the content is correct.

Lubos