Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1113202

Summary: Guests with VDC entitlement can not access content.
Product: [Retired] Subscription Asset Manager Reporter: Michael Stead <mstead>
Component: candlepinAssignee: Michael Stead <mstead>
Status: CLOSED ERRATA QA Contact: SAM QE List <sam-qe-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 1.4CC: bkearney, dgoodwin, liliu, rshutt, shihliu, sthirugn, xdmoon
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-07-17 16:23:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1113173    

Description Michael Stead 2014-06-25 15:47:27 UTC
For VDC subscriptions, the host entitlements have no content. This is what's getting exported to a distributor and stored as the upstream cert. When a guest eventually goes to use an entitlement from it's derived VDC pool, the upstream cert has no content on it, preventing the guest from getting content it is entitled to.


A potential fix we have been discussing is to have VDC certificates generated differently if being granted to a distributor, and include all content for both the parent and the derived pools.

Comment 2 Liushihui 2014-07-04 08:54:46 UTC
Check it on latest RHEL5.11-Server-20140625.0-x86_64(KVM) against SAM-1.4.1-brew, After subscribe the Datacenter pool in the host,  the host entitlements have no content. When a guest eventually goes to use an entitlement from it's derived VDC pool, it has content. But when run "yum repolist", it will pop up 403 forbidden error.

Main Packages:
subscription-manager-1.11.3-6.el5
python-rhsm-1.11.3-3.el5
virt-who-0.9-5.el5
candlepin-0.9.6.3-1.el6sam.noarch
katello-headpin-1.4.3.26-1.el6sam_splice.noarch

Test steps:
In the host:
1. Register to SAM server and attache Datacenter pool
[root@hp-z220-03 libvirt-test-API]# subscription-manager subscribe --pool=8ac200f546fac1fe0146fb7954e20499                                                              
Successfully attached a subscription for: Red Hat Enterprise Linux for Virtual Datacenters, Standard
[root@hp-z220-03 libvirt-test-API]# subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name: Red Hat Enterprise Linux for Virtual Datacenters, Standard
Provides:          
SKU:               RH00002
Contract:          
Account:           
Serial:            8947596129430265220
Pool ID:           8ac200f546fac1fe0146fb7954e20499
Active:            True
Quantity Used:     1
Service Level:     Standard
Service Type:      L1-L3
Status Details:    
Subscription Type: Stackable
Starts:            12/31/2013
Ends:              12/30/2014
System Type:       Physical

2. Check the content, the host entitlements have no content.
[root@hp-z220-03 libvirt-test-API]# rct cat-cert /etc/pki/entitlement/8947596129430265220.pem | more                                                                    

+-------------------------------------------+
        Entitlement Certificate
+-------------------------------------------+

Certificate:
        Path: /etc/pki/entitlement/8947596129430265220.pem
        Version: 3.2
        Serial: 8947596129430265220
        Start Date: 2013-12-31 05:00:00+00:00
        End Date: 2014-12-31 04:59:59+00:00
        Pool ID: 8ac200f546fac1fe0146fb7954e20499

Subject:
        CN: 8ac200f546fac1fe014700938e333558

Issuer:
        C: US
        CN: samserv.redhat.com
        L: Raleigh
        O: SomeOrg
        OU: SomeOrgUnit
        ST: North Carolina


Order:
        Name: Red Hat Enterprise Linux for Virtual Datacenters, Standard
        Number: 
        SKU: RH00002
        Contract: 
        Account: 
        Service Level: Standard
        Service Type: L1-L3
        Quantity: 50
        Quantity Used: 1
        Socket Limit: 2
        RAM Limit:                                                                                                                                                      
        Core Limit:                                                                                                                                                     
        Virt Only: False                                                                                                                                                
        Stacking ID: RH00002                                                                                                                                            
        Warning Period: 0                                                                                                                                               
        Provides Management: False  

On the guest(RHEL6.5):
1. Register to SAM server and attache the Datacenter pool subpool.
[root@6 ~]# subscription-manager subscribe --pool=8ac200f546fac1fe0147009394d03559
Successfully attached a subscription for: Red Hat Enterprise Linux for Virtual Datacenters, Standard
[root@6 ~]# subscription-manager list --consumed                                                                                                                        
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name: Red Hat Enterprise Linux for Virtual Datacenters, Standard
Provides:          Oracle Java (for RHEL Server)
                   Red Hat Developer Toolset (for RHEL Server)
                   Red Hat Software Collections Beta (for RHEL Server)
                   Red Hat Enterprise Linux Server
                   Red Hat Beta
                   Red Hat Software Collections (for RHEL Server)
SKU:               RH00050
Contract:          None
Account:           None
Serial:            1864934834815288107
Pool ID:           8ac200f546fac1fe0147009394d03559
Active:            True
Quantity Used:     1
Service Level:     Standard
Service Type:      L1-L3
Status Details:    
Starts:            12/31/2013
Ends:              12/31/2014
System Type:       Virtual

2. Check the content, the guest entitlements have contents.
[root@6 ~]# rct cat-cert /etc/pki/entitlement/1864934834815288107.pem | more

+-------------------------------------------+
        Entitlement Certificate
+-------------------------------------------+

Certificate:
        Path: /etc/pki/entitlement/1864934834815288107.pem
        Version: 3.2
        Serial: 1864934834815288107
        Start Date: 2013-12-31 05:00:00+00:00
        End Date: 2014-12-31 04:59:59+00:00
        Pool ID: 8ac200f546fac1fe0147009394d03559

Subject:
        CN: 8ac200f547009a19014700a05e6f012b

Issuer:
        C: US
        CN: samserv.redhat.com
        L: Raleigh
        O: SomeOrg
        OU: SomeOrgUnit
        ST: North Carolina

Product:
        ID: 176
        Name: Red Hat Developer Toolset (for RHEL Server)
        Version: 
        Arch: x86_64,x86
        Tags: 

Product:
        ID: 180
        Name: Red Hat Beta
        Version: 
        Arch: x86_64,ppc64,ia64,ppc,s390,x86,s390x

3. Then yum repolist on the guest,it will pop up 403 error as the following
[root@6 ~]# yum repolist
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
https://samserv.redhat.com:8088/content/dist/rhel/server/6/6Server/x86_64/os/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"
Trying other mirror.
repo id                                                  repo name                                                                                                status
rhel-6-server-rpms                                       Red Hat Enterprise Linux 6 Server (RPMs)                                                                 0
rhel-server-dts-6-rpms                                   Red Hat Developer Toolset RPMs for Red Hat Enterprise Linux 6 Server                                     0
rhel-server-dts2-6-rpms                                  Red Hat Developer Toolset 2 RPMs for Red Hat Enterprise Linux 6 Server                                   0
repolist: 0

Comment 3 Devan Goodwin 2014-07-09 14:48:43 UTC
Fix should now be deployed in production, you will need to re-download the manifest however, but after this it should be possible to verify the bug.

Comment 5 Liushihui 2014-07-16 08:01:00 UTC
It still exist on SAM-1.4.1-RHEL-6-20140714.1. 
Check it on RHEL5.11-server-snapshot-1.0-x86_64 against SAM-1.4.1-RHEL-6-20140714.1,do the same test as comment2, it has the same problem as comment2.

Comment 6 Devan Goodwin 2014-07-16 11:19:10 UTC
Liushihui can you confirm you downloaded a new manifest and did not re-use an old one? 

Could you attach the manifest to this bug as well. 

And could we get details on the SAM server and client to login and see if we can debug.

Thanks!

Comment 7 Liushihui 2014-07-16 11:58:54 UTC
I have downloaded the new manifest to SAM server, please check my env as the following:
SAM server:10.66.128.28
KVM host: 10.66.100.108
RHEL6.5 Guest: 10.66.103.160

(In reply to Devan Goodwin from comment #6)
> Liushihui can you confirm you downloaded a new manifest and did not re-use
> an old one? 
> 
> Could you attach the manifest to this bug as well. 
> 
> And could we get details on the SAM server and client to login and see if we
> can debug.
> 
> Thanks!

Comment 8 Devan Goodwin 2014-07-16 12:38:19 UTC
Liushishui: was this a distributor in customer portal that existed before or did you create a new one for this test?

In your manifest I see two datacenter subscriptions, one for premium, and one for standard.

The premium one has content URLs with it, visible in rct cat-manifest. This indicates the fix we implemented is taking place.

The standard one does not seem to have content associated with it.

The guest was using standard and thus was getting denied access because the upstream entitlement did not have any content URLs in it.

I switched the guest to using the premium subscription and was able to successfully use and install content.


So the question is why did that standard subscription not have content? It *could* be caused by an old entitlement attached to the distributor, we can tell for sure if you create a new distributor, or remove that entitlement from the distributor and attach a new one for standard. We attempted to get IT to automatically refresh such entitlements but perhaps something got missed somehow. 

I checked the manifest more closely, the entitlement for datacenter standard support is 8a99f98446d442990146d60555eb053f, it was created on June 26 for SKU RH00002. It's last updated date was also June 26, so this cert does not appear to have been regenerated by IT.

IT supposedly regenerated all affected certificates on July 9th, however this was done in production. Looking in the manifest again, I can see this was from *stage* where we did *not* regenerate any entitlements.



Conclusion, the bug appears to be fixed, if you wish to re-test you will need a newly created distributor in stage environment. Otherwise you run the risk of stale entitlements generated *before* this fix was applied. 

This should not be a problem in production as we explicitly flagged customer entitlements for regeneration if they were affected, so when they get a new manifest the certificate should be correct.

Comment 9 Liushihui 2014-07-17 08:30:16 UTC
Create a new distributor and attach a new datacenter pool, Check it on RHEL5.11-server-snapshot-1.0-x86_64 against SAM-1.4.1-RHEL-6-20140714.1,do the same test as comment2, it can't reproduce now. Therefore, Verify it on SAM-1.4.1-RHEL-6-20140714.1

[root@6 ~]# yum repolist
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
rhel-6-server-rpms                                                                                                   | 3.7 kB     00:00     
rhel-server-dts-6-rpms                                                                                               | 2.9 kB     00:00     
rhel-server-dts2-6-rpms                                                                                              | 2.6 kB     00:00     
repo id                                    repo name                                                                                  status
rhel-6-server-rpms                         Red Hat Enterprise Linux 6 Server (RPMs)                                                   12,663
rhel-server-dts-6-rpms                     Red Hat Developer Toolset RPMs for Red Hat Enterprise Linux 6 Server                           84
rhel-server-dts2-6-rpms                    Red Hat Developer Toolset 2 RPMs for Red Hat Enterprise Linux 6 Server                        380
repolist: 13,127

Comment 10 sthirugn@redhat.com 2014-07-17 14:46:40 UTC
Note: Customer must re-download their manifest after accepting this update.

Comment 12 errata-xmlrpc 2014-07-17 16:23:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0901.html