Bug 859197

Summary: Product ID Cert Deletion Broken Due to Bad Logging Statement
Product: Red Hat Enterprise Linux 6 Reporter: Devan Goodwin <dgoodwin>
Component: subscription-managerAssignee: candlepin-bugs
Status: CLOSED CURRENTRELEASE QA Contact: IDM QE LIST <seceng-idm-qe-list>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 6.5CC: alikins, dgregor, fsharath, jgalipea, jsefler, lnovich
Target Milestone: beta   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-01 13:50:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 840995, 862910, 920191    

Description Devan Goodwin 2012-09-20 19:22:20 UTC 
Description of problem:

product-id plugin has an old reference to a getName method on certificates that is now gone.

Version-Release number of selected component (if applicable):

subscription-manager-1.1.1-1

How reproducible:

Must trigger product cert cleanup which requires a product cert installed where you no longer have anything from it's repos installed.

Steps to Reproduce:
1. Subscribe to a product not yet installed on a system, HA was mentioned when this was raised.
2. Install a package from the new repo. (example: "ccs")
3. yum remove ccs 
  
Actual results:

Will see an error during the yum remove as the product ID plugin tries to clean up the product cert which is no longer providing anything installed.

Expected results:

No errors, cert cleans up normally.

Additional info:

Simple fix, getName should be just property name.
Comment 1 Devan Goodwin 2012-09-21 19:43:16 UTC 
Fixed in subscription-manager.git master: 
06437b86adf1c60283c78268b24bb1751a4a4a57

awood will bring into 5.9 branches.
Comment 2 Devan Goodwin 2012-09-25 14:09:20 UTC 
Pushing out to 6.4, product certs don't get deleted on RHEL5 so we can't test this there.
Comment 6 John Sefler 2013-01-05 14:48:05 UTC 
Testing Version....
DISTRO=RHEL6.4-20130103.n.0
[root@qe-blade-14 ~]# rpm -q subscription-manager python-rhsm yum
subscription-manager-1.1.18-1.el6.x86_64
python-rhsm-1.1.8-1.el6.x86_64
yum-3.2.29-38.el6.noarch

PAY CLOSE ATTENTION TO THE FOLLOWING EVENTS BECAUSE THIS BUG IS NOT ALWAYS REPRODUCIBLE.
(IN MY OPINION, THE FOLLOWING SCENARIO PROVIDES EVIDENCE OF WHY RFE BUG 807762 SHOULD BE CONSIDERED)

[root@qe-blade-14 ~]# ls /etc/pki/product/
69.pem
[root@qe-blade-14 ~]# subscription-manager list --installed
+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:         	Red Hat Enterprise Linux Server
Product ID:           	69
Version:              	6.3
Arch:                 	x86_64
Status:               	Not Subscribed
Starts:               	
Ends:    

[root@qe-blade-14 ~]# yum clean all -q
[root@qe-blade-14 ~]# yum repolist
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
repolist: 0
[root@qe-blade-14 ~]# 
[root@qe-blade-14 ~]# subscription-manager register --username=stage_test_2 --serverurl=subscription.rhn.stage.redhat.com
Password: 
The system has been registered with id: 327298ba-3f62-4590-ab0d-0b0962cb0dc9 
[root@qe-blade-14 ~]# subscription-manager list --avail | grep -i -A2 High-Availability
Subscription Name:    	High-Availability (8 sockets)
SKU:                  	RH1149049
Pool Id:              	8a99f9833c01cc09013c025321d00130
--
Subscription Name:    	High-Availability (8 sockets)
SKU:                  	RH1149049
Pool Id:              	8a99f9843c01ccba013c037a0fa0015a
[root@qe-blade-14 ~]# subscription-manager subscribe --pool=8a99f9833c01cc09013c025321d00130
Successfully attached a subscription for: High-Availability (8 sockets)
[root@qe-blade-14 ~]# 
[root@qe-blade-14 ~]# yum repolist
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
rhel-ha-for-rhel-6-server-rpms                               | 3.7 kB     00:00     
rhel-ha-for-rhel-6-server-rpms/primary_db                    | 160 kB     00:00     
repo id                        repo name                                      status
rhel-ha-for-rhel-6-server-rpms Red Hat Enterprise Linux High Availability (fo 217
repolist: 217
[root@qe-blade-14 ~]# 
[root@qe-blade-14 ~]# yum list available ccs
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
rhel-ha-for-rhel-6-server-rpms                               | 3.7 kB     00:00     
Available Packages
ccs.x86_64               0.16.2-55.el6                rhel-ha-for-rhel-6-server-rpms
[root@qe-blade-14 ~]# ls /etc/pki/product/
69.pem
[root@qe-blade-14 ~]# yum install ccs -y
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
rhel-ha-for-rhel-6-server-rpms                               | 3.7 kB     00:00     
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package ccs.x86_64 0:0.16.2-55.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================
 Package  Arch        Version             Repository                           Size
====================================================================================
Installing:
 ccs      x86_64      0.16.2-55.el6       rhel-ha-for-rhel-6-server-rpms       47 k

Transaction Summary
====================================================================================
Install       1 Package(s)

Total download size: 47 k
Installed size: 263 k
Downloading Packages:
ccs-0.16.2-55.el6.x86_64.rpm                                 |  47 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : ccs-0.16.2-55.el6.x86_64                                         1/1 
rhel-ha-for-rhel-6-server-rpms/productid                     | 1.7 kB     00:00     
  Verifying  : ccs-0.16.2-55.el6.x86_64                                         1/1 

Installed:
  ccs.x86_64 0:0.16.2-55.el6                                                        

Complete!
[root@qe-blade-14 ~]# ls /etc/pki/product/
83.pem
[root@qe-blade-14 ~]# subscription-manager list --installed
+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:         	Red Hat Enterprise Linux High Availability (for RHEL
                        Server)
Product ID:           	83
Version:              	6.3
Arch:                 	x86_64
Status:               	Subscribed
Starts:               	01/03/2013
Ends:                 	01/02/2014


BANG!!!  MY INSTALLED PRODUCT CERT 69.pem FOR RHEL WAS DELETED!  (see trace [1] of rhsm.log below)
WHY DID THIS HAPPEN?  IS HA SUPPOSED TO TRUMP RHEL?  I DON'T THINK SO.
FOLLOWING ARE THE TAGS THAT THE HA PRODUCT CERT PROVIDES (NOTE THAT THEY DO NOT INCLUDE THE RHEL 69.pem TAGS: rhel-6,rhel-6-server)
[root@qe-blade-14 ~]# rct cat-cert /etc/pki/product/83.pem | grep Tags
	Tags: rhel-6-server-highavailability,rhel-6-highavailability


NOW LET'S REMOVE THE ONLY INSTALLED PACKAGE (ccs) FROM REPO rhel-ha-for-rhel-6-server-rpms
AND SEE IF THE YUM product-id PLUGIN DELETES THE PRODUCT CERT 83.pem AS IT IS SUPPOSED TO ON RHEL6...
(ALTHOUGH NOT DEMONSTRATED IN THIS BUG COMMENT, I KNOW THAT ccs IS THE ONLY INSTALLED PACKAGE FROM HA BECAUSE NONE OF THE OTHER PACKAGES IN THIS LISTING ARE INSTALLED http://download.devel.redhat.com/nightly/RHEL6.4-20130103.n.0/6.4/Server/x86_64/os/HighAvailability/listing)


[root@qe-blade-14 ~]# ls /etc/pki/product/
83.pem
[root@qe-blade-14 ~]# yum remove ccs -y
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package ccs.x86_64 0:0.16.2-55.el6 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================
 Package  Arch        Version            Repository                            Size
====================================================================================
Removing:
 ccs      x86_64      0.16.2-55.el6      @rhel-ha-for-rhel-6-server-rpms      263 k

Transaction Summary
====================================================================================
Remove        1 Package(s)

Installed size: 263 k
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing    : ccs-0.16.2-55.el6.x86_64                                         1/1 
  Verifying  : ccs-0.16.2-55.el6.x86_64                                         1/1 

Removed:
  ccs.x86_64 0:0.16.2-55.el6                                                        

Complete!
[root@qe-blade-14 ~]# ls /etc/pki/product/
83.pem
[root@qe-blade-14 ~]# subscription-manager list --installed
+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:         	Red Hat Enterprise Linux High Availability (for RHEL
                        Server)
Product ID:           	83
Version:              	6.3
Arch:                 	x86_64
Status:               	Subscribed
Starts:               	01/03/2013
Ends:                 	01/02/2014


NOPE... REMOVING THE ONLY INSTALLED HIGH-AVAILABILITY PACKAGE DID NOT REMOVE PRODUCT CERT 83.pem AS THE prioduct-id YUM PLUGIN WAS SUPPOSED TO DO.  (see trace [2])

EVEN WORSE....  NOW THAT MY RHEL6 PRODUCT CERT IS GONE, I CAN NO LONGER INSTALL PACKAGES FROM HIGH-AVAILABILITY (SINCE THEIR CONTENT SETS REQUIRE THE RHEL TAGS.   I'M TOTALLY BLOCKED WITHOUT MY RHEL6 PRODUCT CERT AND CANNOT GET IT BACK)...

[root@qe-blade-14 ~]# yum install ccs -y
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
Setting up Install Process
No package ccs available.
Error: Nothing to do
[root@qe-blade-14 ~]# yum list available
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
[root@qe-blade-14 ~]#



Moving back to ASSIGNED/FailedQA

________________________________________________________
[1] [root@qe-blade-14 ~]# tail -f /var/log/rhsm/rhsm.log
2013-01-05 08:37:23,264 [DEBUG]  @profile.py:95 - Loading current RPM profile.
2013-01-05 08:37:23,398 [INFO]  @connection.py:538 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2013-01-05 08:37:23,398 [INFO]  @connection.py:549 - Connection Built: host: subscription.rhn.stage.redhat.com, port: 443, handler: /subscription
2013-01-05 08:37:23,400 [DEBUG]  @connection.py:360 - Loading CA PEM certificates from: /etc/rhsm/ca/
2013-01-05 08:37:23,400 [DEBUG]  @connection.py:342 - Loading CA certificate: '/etc/rhsm/ca/redhat-uep.pem'
2013-01-05 08:37:23,401 [DEBUG]  @connection.py:342 - Loading CA certificate: '/etc/rhsm/ca/candlepin-stage.pem'
2013-01-05 08:37:23,401 [DEBUG]  @connection.py:381 - Making request: GET /subscription/consumers/327298ba-3f62-4590-ab0d-0b0962cb0dc9/release
2013-01-05 08:37:26,119 [DEBUG]  @connection.py:394 - Response status: 200
2013-01-05 08:37:26,219 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-rpms
2013-01-05 08:37:26,219 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-beta-rpms
2013-01-05 08:37:26,219 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-beta-source-rpms
2013-01-05 08:37:26,220 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-beta-debug-rpms
2013-01-05 08:37:26,220 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-rpms
2013-01-05 08:37:26,220 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-debug-rpms
2013-01-05 08:37:26,221 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-beta-rpms
2013-01-05 08:37:26,221 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-source-rpms
2013-01-05 08:37:26,221 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-source-rpms
2013-01-05 08:37:26,221 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-beta-source-rpms
2013-01-05 08:37:26,221 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-debug-rpms
2013-01-05 08:37:26,222 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-beta-debug-rpms
2013-01-05 08:37:26,230 [INFO]  @repolib.py:158 - repos updated: 18
2013-01-05 08:37:30,291 [INFO]  @productid.py:215 - product cert 69 for Red Hat Enterprise Linux Server is being deleted
2013-01-05 08:37:30,291 [INFO]  @productid.py:215 - product cert 69 for Red Hat Enterprise Linux Server is being deleted
2013-01-05 08:37:30,292 [DEBUG]  @productid.py:141 - Updating installed certificates
2013-01-05 08:37:30,292 [DEBUG]  @productid.py:141 - Updating installed certificates
2013-01-05 08:37:30,292 [DEBUG]  @productid.py:143 - product cert: 83 repo: rhel-ha-for-rhel-6-server-rpms
2013-01-05 08:37:30,292 [DEBUG]  @productid.py:143 - product cert: 83 repo: rhel-ha-for-rhel-6-server-rpms
2013-01-05 08:37:30,293 [INFO]  @productid.py:181 - Installed product cert: Red Hat Enterprise Linux High Availability (for RHEL Server) /etc/pki/product/83.pem
2013-01-05 08:37:30,293 [INFO]  @productid.py:181 - Installed product cert: Red Hat Enterprise Linux High Availability (for RHEL Server) /etc/pki/product/83.pem




________________________________________________________
[2] [root@qe-blade-14 ~]# tail -f /var/log/rhsm/rhsm.log
2013-01-05 09:01:02,303 [DEBUG]  @profile.py:95 - Loading current RPM profile.
2013-01-05 09:01:02,440 [INFO]  @connection.py:538 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2013-01-05 09:01:02,441 [INFO]  @connection.py:549 - Connection Built: host: subscription.rhn.stage.redhat.com, port: 443, handler: /subscription
2013-01-05 09:01:02,442 [DEBUG]  @connection.py:360 - Loading CA PEM certificates from: /etc/rhsm/ca/
2013-01-05 09:01:02,443 [DEBUG]  @connection.py:342 - Loading CA certificate: '/etc/rhsm/ca/redhat-uep.pem'
2013-01-05 09:01:02,443 [DEBUG]  @connection.py:342 - Loading CA certificate: '/etc/rhsm/ca/candlepin-stage.pem'
2013-01-05 09:01:02,444 [DEBUG]  @connection.py:381 - Making request: GET /subscription/consumers/327298ba-3f62-4590-ab0d-0b0962cb0dc9/release
2013-01-05 09:01:03,199 [DEBUG]  @connection.py:394 - Response status: 200
2013-01-05 09:01:03,299 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-6-server', skipping content: rhel-ha-for-rhel-6-server-debug-rpms
2013-01-05 09:01:03,299 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-rpms
2013-01-05 09:01:03,299 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-beta-rpms
2013-01-05 09:01:03,299 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-beta-source-rpms
2013-01-05 09:01:03,299 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-6-server', skipping content: rhel-ha-for-rhel-6-server-rpms
2013-01-05 09:01:03,300 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-beta-debug-rpms
2013-01-05 09:01:03,300 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-rpms
2013-01-05 09:01:03,300 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-6-server', skipping content: rhel-ha-for-rhel-6-server-source-rpms
2013-01-05 09:01:03,300 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-6-server', skipping content: rhel-ha-for-rhel-6-server-beta-debug-rpms
2013-01-05 09:01:03,300 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-debug-rpms
2013-01-05 09:01:03,300 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-6-server', skipping content: rhel-ha-for-rhel-6-server-beta-rpms
2013-01-05 09:01:03,301 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-beta-rpms
2013-01-05 09:01:03,301 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-source-rpms
2013-01-05 09:01:03,301 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-source-rpms
2013-01-05 09:01:03,301 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-beta-source-rpms
2013-01-05 09:01:03,301 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-ibm-power', skipping content: rhel-ha-for-rhel-5-for-power-debug-rpms
2013-01-05 09:01:03,301 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-6-server', skipping content: rhel-ha-for-rhel-6-server-beta-source-rpms
2013-01-05 09:01:03,301 [DEBUG]  @repolib.py:197 - Missing required tag 'rhel-5-server', skipping content: rhel-ha-for-rhel-5-server-beta-debug-rpms
2013-01-05 09:01:03,302 [INFO]  @repolib.py:158 - repos updated: 6
2013-01-05 09:01:04,906 [DEBUG]  @productid.py:141 - Updating installed certificates
2013-01-05 09:01:04,906 [DEBUG]  @productid.py:141 - Updating installed certificates
Comment 7 Devan Goodwin 2013-01-07 16:48:43 UTC 
Consistent reproducer, start with a plain RHEL Server system, and copy /etc/pki/product/69.pem as well as /var/lib/rhsm/productid.js somewhere safe.

productid.js should look something like this:

[root@localhost ~]# cat /var/lib/rhsm/productid.js 
{
  "69": "anaconda-RedHatEnterpriseLinux-201211201732.x86_64"
}

Now reproduce using steps above.

To reset the machine:

1. unregister
2. Remove any product certs in /etc/pki/product.
3. Restore the two files you backed up.
4. Make sure ccs is uninstalled.
5. yum clean all

You should now be able to re-try the scenario above and get the same errors.
Comment 8 Devan Goodwin 2013-01-07 18:05:22 UTC 
This does not appear to be a regression, just a new scenario that has never been handled. Bug seems to be reproducible with 6.3 subman and python-rhsm.

After install we have an entry in the productid.js database for 69 pointing to an anaconda repo from installation. This is not actually a yum repo anymore, nothing in config after install.

If you subscribe first to something that does not provide your installed base product, subscription manager does not see that this anaconda repo is active (as it doesn't exist) and cleans up product 69, there's no way for us to know that this product ID is somehow special and should not be cleaned when nothing is installed from it. (we don't even have a real repo to check against)

bkearney has suggested that we tackle this with some upcoming branding changes where certain product IDs will be known to be special. We should be able to leverage this to know when to protect a product cert.

Due to not being a regression we're going to push to RHEL7 tracker and lower priority.
Comment 10 RHEL Program Management 2013-01-13 06:47:17 UTC 
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 13 Adrian Likins 2013-03-15 20:08:42 UTC 
(eh, comment 12 is incomplete)

For rhel, one of the questions is how do we determine if a product cert should not be deleted. In these cases, the rhel product cert meets all the current requirements for being deleted:

1. product cert is installed
2. the product cert-> repo database (/var/lib/rhsm/productid.js) has an entry
for the product id
3. There is a repo for that product
4. that repo is enabled
5. the repo has correct metadata info (aka, a product id)
6. no packages are installed from that repo (they are all from 'anaconda' repo)

That normally means "we dont need that product cert anymore, delete it."
Comment 15 Adrian Likins 2013-03-26 14:06:05 UTC 
commit ff4c9ab4a4785cfb2849766e20f6805b3ed06c96
Merge: 7281b51 1249eca
Author: Devan Goodwin <dgoodwin>
Date:   Tue Mar 26 06:14:17 2013 -0700

    Merge pull request #542 from candlepin/alikins/product_id
    
    Alikins/product_id


Fix in summation:

tl;dr:  hardcode to not delete rhel product certs


More specifically, do not delete product certs that have provide tags of the
form 'rhel*'. 

There are actually a lot of other changes, fixed, and improvements in that merge that fix a lot of other odd/weird/broken behavior in product cert installation and deletion. But none of those fixed the particular case of:

- rhel installed by anaconda
- rhel product cert installed by anaconda from the anaconda repo
- system register
- system entitled to rhel-6 content
- NO PACKAGES INSTALLED FROM THAT RHEL-6 CONTENT REPO
- other repo's enabled (could be rhsm repos, or other)
- yum install 'some-package-not-from-rhel-repo'
  (ie, rhel-6 repo is 'enabled', but not 'active' [no installed packages come from it], and we do something that get's us into the product cert installtion code (installing a package)

Other changes fixes:

- productid.js is now of the format:

     {'69': ['repo-1', 'repo-2]}

 instead of:
   
      {'69': 'repo-1'}

  We now track more than one repo per product cert, so we can track if a product
  id cert was installed from one repo, but is now in another repo. Or if 
  product cert was found in multiple repos. 

  This helps track cases where repo id's change and multiple repos with same
  product cert.

- lots of refactoring of code paths, and inline documentation included
- change code to find all product certs to be deleted or installed first, then
  install/delete them at once. This simplifies some corner cases, and provides 
  better data to plugin hooks.

- if a product cert is found in a repo, and we don't track the repo it came 
  from, add the repo id to the list of repos that provide that id, instead of
  dropping that info, or replacing existing info. This also makes the code
  that determines if a product id cert has been installed more accurate.
Comment 16 RHEL Program Management 2013-04-09 20:01:36 UTC 
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux release.  Product Management has
requested further review of this request by Red Hat Engineering, for
potential inclusion in a Red Hat Enterprise Linux release for currently
deployed products.  This request is not yet committed for inclusion in
a release.
Comment 19 John Sefler 2013-06-07 16:16:07 UTC 
Testing Version....
[root@rhsm-compat-rhel64 ~]# rpm -q subscription-manager python-rhsm
subscription-manager-1.8.10-1.el6.x86_64
python-rhsm-1.8.12-1.el6.x86_64

[root@rhsm-compat-rhel64 ~]# ls /etc/pki/product/
69.pem
[root@rhsm-compat-rhel64 ~]# subscription-manager list --installed
+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux Server
Product ID:     69
Version:        6.3
Arch:           x86_64
Status:         Unknown
Status Details: 
Starts:         
Ends:           

[root@rhsm-compat-rhel64 ~]# cat /var/lib/rhsm/productid.js
{
  "69": "anaconda-RedHatEnterpriseLinux-201301162040.x86_64"
}

[root@rhsm-compat-rhel64 ~]# yum clean all -q
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
[root@rhsm-compat-rhel64 ~]# yum repolist
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security,
              : subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
repolist: 0
[root@rhsm-compat-rhel64 ~]# subscription-manager register --username=stage_test_2 --serverurl=subscription.rhn.stage.redhat.com
Password: 
The system has been registered with ID: 5efba2fb-5764-44a4-a848-1df2cc90529e 
[root@rhsm-compat-rhel64 ~]# subscription-manager list --avail | grep -i -A2 High-Availability
Subscription Name: High-Availability (8 sockets)
SKU:               RH1149049
Pool ID:           8a99f9843c01ccba013c037a0fd40169
[root@rhsm-compat-rhel64 ~]# subscription-manager subscribe --pool 8a99f9843c01ccba013c037a0fd40169
Successfully attached a subscription for: High-Availability (8 sockets)
[root@rhsm-compat-rhel64 ~]# 
[root@rhsm-compat-rhel64 ~]# yum repolist
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security,
              : subscription-manager
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
rhel-ha-for-rhel-6-server-rpms                           | 3.7 kB     00:00     
rhel-ha-for-rhel-6-server-rpms/primary_db                | 184 kB     00:00     
repo id                        repo name                                  status
rhel-ha-for-rhel-6-server-rpms Red Hat Enterprise Linux High Availability 258
repolist: 258
[root@rhsm-compat-rhel64 ~]# 
[root@rhsm-compat-rhel64 ~]# yum list available ccs
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security,
              : subscription-manager
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
rhel-ha-for-rhel-6-server-rpms                           | 3.7 kB     00:00     
Available Packages
ccs.x86_64             0.16.2-63.el6              rhel-ha-for-rhel-6-server-rpms
[root@rhsm-compat-rhel64 ~]# 
[root@rhsm-compat-rhel64 ~]# ls /etc/pki/product/
69.pem
[root@rhsm-compat-rhel64 ~]# cat /var/lib/rhsm/productid.js
{
  "69": "anaconda-RedHatEnterpriseLinux-201301162040.x86_64"
}
[root@rhsm-compat-rhel64 ~]# 
[root@rhsm-compat-rhel64 ~]# yum install ccs -y
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security,
              : subscription-manager
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
rhel-ha-for-rhel-6-server-rpms                           | 3.7 kB     00:00     
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package ccs.x86_64 0:0.16.2-63.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package Arch       Version            Repository                          Size
================================================================================
Installing:
 ccs     x86_64     0.16.2-63.el6      rhel-ha-for-rhel-6-server-rpms      48 k

Transaction Summary
================================================================================
Install       1 Package(s)

Total download size: 48 k
Installed size: 285 k
Downloading Packages:
ccs-0.16.2-63.el6.x86_64.rpm                             |  48 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
  Installing : ccs-0.16.2-63.el6.x86_64                                     1/1 
rhel-ha-for-rhel-6-server-rpms/productid                 | 1.7 kB     00:00     
  Verifying  : ccs-0.16.2-63.el6.x86_64                                     1/1 

Installed:
  ccs.x86_64 0:0.16.2-63.el6                                                    

Complete!
[root@rhsm-compat-rhel64 ~]# 
[root@rhsm-compat-rhel64 ~]# ls /etc/pki/product/
69.pem  83.pem
[root@rhsm-compat-rhel64 ~]# cat /var/lib/rhsm/productid.js
{
  "69": "anaconda-RedHatEnterpriseLinux-201301162040.x86_64", 
  "83": [
    "rhel-ha-for-rhel-6-server-rpms"
  ]
}
[root@rhsm-compat-rhel64 ~]# subscription-manager list --installed
+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux High Availability (for RHEL Server)
Product ID:     83
Version:        6.3
Arch:           x86_64
Status:         Subscribed
Status Details: 
Starts:         12/31/2012
Ends:           12/31/2013

Product Name:   Red Hat Enterprise Linux Server
Product ID:     69
Version:        6.3
Arch:           x86_64
Status:         Not Subscribed
Status Details: High 
Starts:         
Ends:           


VERIFIED! My previously installed RHEL product cert 69 remained installed and unsubscribed while a new product certs for High Availablility was installed and subscribed!  That's what we wanted.

NOW LET'S REMOVE THE ONLY INSTALLED PACKAGE (ccs) FROM REPO rhel-ha-for-rhel-6-server-rpms
AND SEE IF THE YUM product-id PLUGIN DELETES THE PRODUCT CERT 83.pem AS IT IS SUPPOSED TO ON RHEL6...

[root@rhsm-compat-rhel64 ~]# yum remove ccs -y
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security,
              : subscription-manager
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package ccs.x86_64 0:0.16.2-63.el6 will be erased
--> Finished Dependency Resolution
rhel-ha-for-rhel-6-server-rpms                           | 3.7 kB     00:00     

Dependencies Resolved

================================================================================
 Package Arch       Version           Repository                           Size
================================================================================
Removing:
 ccs     x86_64     0.16.2-63.el6     @rhel-ha-for-rhel-6-server-rpms     285 k

Transaction Summary
================================================================================
Remove        1 Package(s)

Installed size: 285 k
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing    : ccs-0.16.2-63.el6.x86_64                                     1/1 
  Verifying  : ccs-0.16.2-63.el6.x86_64                                     1/1 

Removed:
  ccs.x86_64 0:0.16.2-63.el6                                                    

Complete!
[root@rhsm-compat-rhel64 ~]# ls /etc/pki/product/
69.pem  83.pem

NOPE!  product 83.pem should have been removed since ccs was the only package installed from @rhel-ha-for-rhel-6-server-rpms  The following yum list installed shows that no packages remain from repo rhel-ha-for-rhel-6-server-rpms
[root@rhsm-compat-rhel64 ~]# yum list installed | grep rhel-ha-for-rhel-6
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
[root@rhsm-compat-rhel64 ~]# 

HOWEVER,  As decided in comment 15, we will "not delete product certs that have provide tags of the
form 'rhel*'."  83.pem does provide tags of the form rhel*...
[root@rhsm-compat-rhel64 ~]# rct cat-cert /etc/pki/product/83.pem | grep Tags
	Tags: rhel-6-server-highavailability,rhel-6-highavailability
[root@rhsm-compat-rhel64 ~]# 


For the fun of it, let's subscribe to RHEL and see what happens to our rhel 69 product cert...


[root@rhsm-compat-rhel64 ~]# subscription-manager list --avail | grep RH0103708 -B2 -A1
Subscription Name: Red Hat Enterprise Linux Server, Premium (8 sockets) (Up to 4
                   guests)
SKU:               RH0103708
Pool ID:           8a99f9833c01cc09013c02532185010e

[root@rhsm-compat-rhel64 ~]# subscription-manager subscribe --pool 8a99f9843c01ccba013c037a1035018b
Successfully attached a subscription for: Red Hat Enterprise Linux Server, Premium (8 sockets) (Up to 4 guests)
[root@rhsm-compat-rhel64 ~]# yum install zsh -q -y
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
[root@rhsm-compat-rhel64 ~]# yum list installed zsh
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security,
              : subscription-manager
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
rhel-6-server-cf-tools-1-rpms                            | 2.8 kB     00:00     
rhel-6-server-rhev-agent-rpms                            | 3.1 kB     00:00     
rhel-6-server-rpms                                       | 3.7 kB     00:00     
rhel-ha-for-rhel-6-server-rpms                           | 3.7 kB     00:00     
Installed Packages
zsh.x86_64                   4.3.10-5.el6                    @rhel-6-server-rpms
[root@rhsm-compat-rhel64 ~]# cat /var/lib/rhsm/productid.js
{
  "69": [
    "anaconda-RedHatEnterpriseLinux-201301162040.x86_64", 
    "rhel-6-server-cf-tools-1-rpms", 
    "rhel-6-server-rhev-agent-rpms", 
    "rhel-6-server-rpms"
  ], 
  "83": [
    "rhel-ha-for-rhel-6-server-rpms"
  ]
}

VERIFIED:  New implementation keeps an array of all the possibly enitlemed repos that the product cert was installed from.

[root@rhsm-compat-rhel64 ~]# yum remove -q zsh
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.

================================================================================
 Package    Arch          Version              Repository                  Size
================================================================================
Removing:
 zsh        x86_64        4.3.10-5.el6         @        4.8 M

Transaction Summary
================================================================================
Remove        1 Package(s)

Is this ok [y/N]: y
[root@rhsm-compat-rhel64 ~]# yum list installed | grep rhel-6-server
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
python-httplib2.noarch                0.6.0-4.el6_0            @rhel-6-server-cf-tools-1-rpms
rhevm-guest-agent.x86_64              1.0.5-8.el6ev            @rhel-6-server-rhev-agent-rpms
[root@rhsm-compat-rhel64 ~]# cat /var/lib/rhsm/productid.js
{
  "69": [
    "anaconda-RedHatEnterpriseLinux-201301162040.x86_64", 
    "rhel-6-server-cf-tools-1-rpms", 
    "rhel-6-server-rhev-agent-rpms", 
    "rhel-6-server-rpms"
  ], 
  "83": [
    "rhel-ha-for-rhel-6-server-rpms"
  ]
}
[root@rhsm-compat-rhel64 ~]# ls /etc/pki/product
69.pem  83.pem

VERIFIED: despite the removal of zsh (last package from repo rhel-6-server-rpms), the product cert 69 remains installed because other packages are installed from repos that 69 could have come from.


Let's try re-registering to install a non-rhel product...
[root@rhsm-compat-rhel64 ~]# subscription-manager register --username stage_test_47 --force
The system with UUID a19c19ed-286d-4b9f-920d-bb4e28e7dca5 has been unregistered
Password: 
The system has been registered with ID: 9a024bcd-c34d-4a9b-802e-56632af9d70a 
[root@rhsm-compat-rhel64 ~]# subscription-manager list --avail | grep MCT2358 -B1 -A1
Subscription Name: CloudForms (10-pack)
SKU:               MCT2358
Pool ID:           8a99f9843c01ccba013c037f9a22050c
[root@rhsm-compat-rhel64 ~]# subscription-manager subscribe --pool 8a99f9843c01ccba013c037f9a22050c
Successfully attached a subscription for: CloudForms (10-pack)
[root@rhsm-compat-rhel64 ~]# yum list available --enablerepo=rhel-6-server-cf-ce-1-rpms | tail -1
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
unittest.x86_64                       0.50-62.6.el6   rhel-6-server-cf-ce-1-rpms
[root@rhsm-compat-rhel64 ~]# ls /etc/pki/product
69.pem  83.pem
[root@rhsm-compat-rhel64 ~]# yum install unittest --enablerepo=rhel-6-server-cf-ce-1-rpms
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security,
              : subscription-manager
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
rhel-6-server-cf-ce-1-rpms                               | 3.1 kB     00:00     
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package unittest.x86_64 0:0.50-62.6.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package      Arch       Version           Repository                      Size
================================================================================
Installing:
 unittest     x86_64     0.50-62.6.el6     rhel-6-server-cf-ce-1-rpms      37 k

Transaction Summary
================================================================================
Install       1 Package(s)

Total download size: 37 k
Installed size: 141 k
Is this ok [y/N]: y
Downloading Packages:
unittest-0.50-62.6.el6.x86_64.rpm                        |  37 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : unittest-0.50-62.6.el6.x86_64                                1/1 
  Verifying  : unittest-0.50-62.6.el6.x86_64                                1/1 

Installed:
  unittest.x86_64 0:0.50-62.6.el6                                               

Complete!
[root@rhsm-compat-rhel64 ~]# ls /etc/pki/product167.pem  69.pem  83.pem
[root@rhsm-compat-rhel64 ~]# cat /var/lib/rhsm/productid.js
{
  "69": [
    "anaconda-RedHatEnterpriseLinux-201301162040.x86_64", 
    "rhel-6-server-cf-tools-1-rpms", 
    "rhel-6-server-rhev-agent-rpms", 
    "rhel-6-server-rpms"
  ], 
  "167": [
    "rhel-6-server-cf-ce-1-rpms"
  ], 
  "83": [
    "rhel-ha-for-rhel-6-server-rpms"
  ]
}
[root@rhsm-compat-rhel64 ~]# rct cat-cert /etc/pki/product/167.pem | grep Tags
	Tags: None
[root@rhsm-compat-rhel64 ~]#

NOTE THAT PRODUCT CERT 167 DOES NOT PROVIDE rhel* TAGS.

[root@rhsm-compat-rhel64 ~]# yum list installed | grep rhel-6-server-cf-ce-1-rpms
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
unittest.x86_64                       0.50-62.6.el6            @rhel-6-server-cf-ce-1-rpms
[root@rhsm-compat-rhel64 ~]# yum remove unittest
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security,
              : subscription-manager
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package unittest.x86_64 0:0.50-62.6.el6 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package     Arch      Version             Repository                      Size
================================================================================
Removing:
 unittest    x86_64    0.50-62.6.el6       @rhel-6-server-cf-ce-1-rpms    141 k

Transaction Summary
================================================================================
Remove        1 Package(s)

Installed size: 141 k
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing    : unittest-0.50-62.6.el6.x86_64                                1/1 
  Verifying  : unittest-0.50-62.6.el6.x86_64                                1/1 

Removed:
  unittest.x86_64 0:0.50-62.6.el6                                               

Complete!
[root@rhsm-compat-rhel64 ~]# ls /etc/pki/product
167.pem  69.pem  83.pem
[root@rhsm-compat-rhel64 ~]# cat /var/lib/rhsm/productid.js
{
  "69": [
    "anaconda-RedHatEnterpriseLinux-201301162040.x86_64", 
    "rhel-6-server-cf-tools-1-rpms", 
    "rhel-6-server-rhev-agent-rpms", 
    "rhel-6-server-rpms"
  ], 
  "167": [
    "rhel-6-server-cf-ce-1-rpms"
  ], 
  "83": [
    "rhel-ha-for-rhel-6-server-rpms"
  ]
}
[root@rhsm-compat-rhel64 ~]# ls /etc/pki/product
167.pem  69.pem  83.pem
[root@rhsm-compat-rhel64 ~]# 

BANG! Despite the fact that we removed the only package (unittest) remaining from repo rhel-6-server-cf-ce-1-rpms whose original installation gave us product cert 167 (which does NOT provide "rhel*" tags), product cert 167 remains installed on the system.  I believe this is wrong.  I believe product cert 167 should have been removed.


Moving This bug to VERIFIED because the original problem is fixed.  Opening new bug 971945 to address the failure to remove the product cert upon yum removal of the final package from an entitled repo.