Bug 721012

Summary: tracking hosted issue: Unable to upload content certs generated with stage environment.
Product: Red Hat Update Infrastructure for Cloud Providers Reporter: Sachin Ghai <sghai>
Component: RHUAAssignee: Jay Dobies <jason.dobies>
Status: CLOSED CURRENTRELEASE QA Contact: wes hayutin <whayutin>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 2.0CC: acarter, kbidarka, sghai, tsanders
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-05-31 12:57:45 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Sachin Ghai 2011-07-13 14:31:26 UTC 
Description of problem:

We got the access to stage environment (https://access.stage.redhat.com/management/) to generate the content certs. I generated the certs using subscription manager.

[root@dhcp201-141 ~]# cd /etc/pki/entitlement/
[root@dhcp201-141 entitlement]# ls
7177615796826081838.pem  key.pem  stage_content.pem
[root@dhcp201-141 entitlement]# 

Here I combined both key and certs in one file named as "stage_content.pem".

When I tried to upload in rhui-manager, i got following:


------------------------------------------------------------------------------
rhui (entitlements) => u

Full path to the new content certificate:
/etc/pki/entitlement/stage_content.pem

The RHUI will be updated with the following certificate:
  /etc/pki/entitlement/stage_content.pem

Proceed? (y/n) y

The given certificate contains one or more entitlements that are
not compatible with the RHUI. For questions, please visit:
https://access.redhat.com/support/contact/customerService.html

------------------------------------------------------------------------------
rhui (entitlements) => 



Version-Release number of selected component (if applicable):
rhui-tools 2.0.37

How reproducible:
always

Steps to Reproduce:

  
Actual results:
The given certificate contains one or more entitlements that are
not compatible with the RHUI

Expected results:
Generated content certs from stage should be uploaded in rhui successfully.

Additional info:

I opened this bug, just to ensure that we can upload the content certs in rhui, generated through stage env.
Comment 1 wes hayutin 2011-07-13 15:15:13 UTC 
http://qe-india.pad.engineering.redhat.com/37?
Comment 2 Sachin Ghai 2011-07-14 11:43:32 UTC 
Generated new certs and tried to upload them using rhui-manager.

Certs generated under /etc/pki/entitlement directory. 

[root@dhcp201-141 redhat]# cd /etc/pki/entitlement/
[root@dhcp201-141 entitlement]# ll
total 44
-rw-r--r--. 1 root root 17422 Jul 14 06:26 5235838272810302347.pem
-rw-r--r--. 1 root root  1675 Jul 14 06:26 key.pem
-rw-r--r--. 1 root root 19097 Jul 14 16:20 stage_content.pem


First I uploaded the 5235838272810302347.pem file using rhui-manager. All certs uploaded successfuly. See the listing below:

rhui (entitlements) => u

Full path to the new content certificate:
/etc/pki/entitlement/5235838272810302347.pem

The RHUI will be updated with the following certificate:
  /etc/pki/entitlement/5235838272810302347.pem

Proceed? (y/n) y

Updating repository Red Hat Update Infrastructure 1.1 (RPMs) (5Server-i386) with certificate new_content.pem...
... successfully updated
Updating repository Red Hat Update Infrastructure 1.1 (RPMs) (5Server-x86_64) with certificate new_content.pem...
... successfully updated
Updating repository Red Hat Update Infrastructure 1.2 (RPMs) (5Server-i386) with certificate new_content.pem...
... successfully updated
Updating repository Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64) with certificate new_content.pem...
... successfully updated
Updating repository Red Hat Enterprise Linux Server 6 Optional Updates (RPMs) (6Server-x86_64) with certificate new_content.pem...
... successfully updated
Updating repository Red Hat Enterprise Linux Server 6 Updates (RPMs) (6Server-x86_64) with certificate new_content.pem...
... successfully updated

Red Hat Entitlements

  Valid
    Red Hat Enterprise Linux 5 Server Beta from RHUI (Debug RPMs)
    Expiration: 07-11-2012     Certificate: 5235838272810302347.pem

    Red Hat Enterprise Linux 5 Server Beta from RHUI (RPMs)
    Expiration: 07-11-2012     Certificate: 5235838272810302347.pem

    Red Hat Enterprise Linux 5 Server Beta from RHUI (Source RPMs)
    Expiration: 07-11-2012     Certificate: 5235838272810302347.pem

    Red Hat Enterprise Linux 5 Server from RHUI (Debug RPMs)
    Expiration: 07-11-2012     Certificate: 5235838272810302347.pem
    
     : :::    
     
    Red Hat Update Infrastructure 1.2 (Debug RPMs)
    Expiration: 07-11-2012     Certificate: 5235838272810302347.pem

    Red Hat Update Infrastructure 1.2 (ISOs)
    Expiration: 07-11-2012     Certificate: 5235838272810302347.pem

    Red Hat Update Infrastructure 1.2 (RPMs)
    Expiration: 07-11-2012     Certificate: 5235838272810302347.pem

    Red Hat Update Infrastructure 1.2 (RPMs)
    Expiration: 03-21-2012     Certificate: new_content.pem

    Red Hat Update Infrastructure 1.2 (Source RPMs)
    Expiration: 07-11-2012     Certificate: 5235838272810302347.pem

    Red Hat Update Infrastructure i386 Beta Optional (RPMs)
    Expiration: 03-21-2012     Certificate: new_content.pem

    Red Hat Update Infrastructure x86_64 Beta Optional (RPMs)
    Expiration: 03-21-2012     Certificate: new_content.pem

   Expiration: 03-21-2012     Certificate: new_content.pem


------------------------------------------------------------------------------
rhui (entitlements) => exit
Comment 3 Sachin Ghai 2011-07-14 11:45:44 UTC 
However after uploading the content certs ( generated through stage env), I got following in rhui.log

Successfully connected to [dhcp201-141.englab.pnq.redhat.com]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.1.148.1]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.11]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.10]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.13]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.15]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.14]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.5]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.7]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.6]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.1]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.3]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.2]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.5.1]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.1.147.1]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.12]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.16]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.1.148.1]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.11]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.10]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.13]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.15]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.14]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.5]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.7]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.6]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.1]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.3]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.2]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.5.1]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.1.147.1]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.12]
Unexpected OID encountered [1.3.6.1.4.1.2312.9.4.16]
Comment 4 Sachin Ghai 2011-07-14 11:49:02 UTC 
When I tried to add a new repo using rhui-manager, I got following traceback in rhui.log: 
                
               
------------------------------------------------------------------------------
rhui (repo) => a

Loading latest entitled products from Red Hat...

An unexpected error has occurred during the last operation.
More information can be found in /root/.rhui/rhui.log.


------------------------------------------------------------------------------


rhui.log:
=========
Unexpected error caught at the shell level
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/rhui/tools/shell.py", line 85, in safe_listen
    self.listen(clear=first_run)
  File "/usr/lib/python2.6/site-packages/rhui/tools/shell.py", line 111, in listen
    Shell.listen(self)
  File "/usr/lib/python2.6/site-packages/rhui/common/shell.py", line 191, in listen
    item.func(*args, **item.kwargs)
  File "/usr/lib/python2.6/site-packages/rhui/tools/screens/repo.py", line 122, in add
    self.candidate_repo_manager.translate_entitlements()
  File "/usr/lib/python2.6/site-packages/rhui/tools/repo_candidates.py", line 62, in translate_entitlements
    mappings = self.cdn_api.expand_variables(e.download_url, cert.cert_filename)
  File "/usr/lib/python2.6/site-packages/rhui/tools/cdn_api.py", line 70, in expand_variables
    mappings = self._translate_next_variable({'' : url}, cert_filename)
  File "/usr/lib/python2.6/site-packages/rhui/tools/cdn_api.py", line 104, in _translate_next_variable
    substitutions = self._request_get(listing_url, cert_filename).split('\n')[:-1]
  File "/usr/lib/python2.6/site-packages/rhui/tools/cdn_api.py", line 150, in _request_get
    server = self._server(cert_filename)
  File "/usr/lib/python2.6/site-packages/rhui/tools/cdn_api.py", line 176, in _server
    context.load_cert(cert_filename)
  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Context.py", line 79, in load_cert
    m2.ssl_ctx_use_privkey(self.ctx, keyfile)
SSLError: no start line
Comment 5 Sachin Ghai 2011-07-14 12:03:59 UTC 
Then I combined both files (cert and key file) in stage_content.pem as per following cmd and upload it using rhui-manager. 

#cat 5235838272810302347.pem key.pem > stage_content.pem

I got same error in rhui.log saying "Unexpected OID encountered".

However, when I tried to add a new repo, I got following traceback in pulp.log:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/rhui/tools/shell.py", line 85, in safe_listen
    self.listen(clear=first_run)
  File "/usr/lib/python2.6/site-packages/rhui/tools/shell.py", line 111, in listen
    Shell.listen(self)
  File "/usr/lib/python2.6/site-packages/rhui/common/shell.py", line 191, in listen
    item.func(*args, **item.kwargs)
  File "/usr/lib/python2.6/site-packages/rhui/tools/screens/repo.py", line 122, in add
    self.candidate_repo_manager.translate_entitlements()
  File "/usr/lib/python2.6/site-packages/rhui/tools/repo_candidates.py", line 62, in translate_entitlements
    mappings = self.cdn_api.expand_variables(e.download_url, cert.cert_filename)
  File "/usr/lib/python2.6/site-packages/rhui/tools/cdn_api.py", line 70, in expand_variables
    mappings = self._translate_next_variable({'' : url}, cert_filename)
  File "/usr/lib/python2.6/site-packages/rhui/tools/cdn_api.py", line 104, in _translate_next_variable
    substitutions = self._request_get(listing_url, cert_filename).split('\n')[:-1]
  File "/usr/lib/python2.6/site-packages/rhui/tools/cdn_api.py", line 155, in _request_get
    raise Exception(response.status, response.read())
Exception: (404, 'File not found.')
Comment 6 wes hayutin 2011-07-14 19:27:41 UTC 
jdob, I know you guys looked at this bug in the morning, can you please summarize where we are w/ this bug.
Comment 7 Jay Dobies 2011-07-15 14:09:35 UTC 
The "Unexpected OID" thing isn't an error. It's meant as a warning that the certificate is containing data we're not expecting, but we're just going to ignore it.

The 404 comes from the fact that there's an entitlement for:

/content/dist/rhel/rhui/server/6/$releasever/$basearch/source/SRPMS

When we encounter $releasever, we try to look up a listing file in CDN that indicates what to substitute in:

https://cdn.redhat.com/content/dist/rhel/rhui/server/6/listing

If you look in:

https://cdn.redhat.com/content/dist/rhel/rhui/server

There's not even a directory named 6.

So either:
1.  CDN is missing directories.
2.  RHSM web is returning faulty entitlements that indicate content URLs that don't exist.
Comment 8 Jay Dobies 2011-07-15 14:12:09 UTC 
<dgregor> jdob, I'll fix
<jdob> ok, where is the issue, is CDN missing stuff or is RHSM web returning faulty entitlements?
<dgregor> CDN missing stuff
Comment 9 Jay Dobies 2011-07-15 14:23:27 UTC 
All of the content should be in CDN now.
Comment 10 Sachin Ghai 2011-07-18 11:07:23 UTC 
I generated new content certs from stage and combined both the cert and key file in the stage_content.pem as below:

 
root@dhcp201-133 entitlement]# ls
3699916758563663621.pem  key.pem
[root@dhcp201-133 entitlement]# cat 3699916758563663621.pem key.pem > stage_content.pem
[root@dhcp201-133 entitlement]# rhui-manager 


Uploaded the stage_content.pem file using rhui-manager:

------------------------------------------------------------------------------
rhui (entitlements) => u

Full path to the new content certificate:
/root/entitlement/stage_content.pem

The RHUI will be updated with the following certificate:
  /root/entitlement/stage_content.pem

Proceed? (y/n) y


Red Hat Entitlements

  Valid
    Red Hat Enterprise Linux 5 Server Beta from RHUI (Debug RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 5 Server Beta from RHUI (RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 5 Server Beta from RHUI (Source RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 5 Server from RHUI (Debug RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 5 Server from RHUI (RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 5 Server from RHUI (Source RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server - Optional Beta from RHUI (RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server - Optional Beta from RHUI (Source RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server - Optional Beta from RHUI(Debug RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server - Optional from RHUI (Debug RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server - Optional from RHUI (RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server - Optional from RHUI (Source RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server Beta from RHUI (Debug RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server Beta from RHUI (RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server Beta from RHUI (Source RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server from RHUI (Debug RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server from RHUI (RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Enterprise Linux 6 Server from RHUI (Source RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Update Infrastructure 1.2 (Debug RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Update Infrastructure 1.2 (ISOs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Update Infrastructure 1.2 (RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem

    Red Hat Update Infrastructure 1.2 (Source RPMs)
    Expiration: 07-11-2012     Certificate: stage_content.pem


------------------------------------------------------------------------------
rhui (entitlements) => ^



This time I didn't get any "oid" related error in rhui.log. I added a new repo with stage entitlements, following is the repo.

Red Hat Enterprise Linux 6 Server - Optional from RHUI (Source RPMs) 

Repo was added successfuly without any error in pulp.log
Comment 11 Sachin Ghai 2011-07-18 12:36:27 UTC 
I started the sync for newly addded repo. 

And later uploaded old content certs ( rhui-amazon). This time again I added following two repos using old content certs.

- Red Hat Update Infrastructure 1.2 (RPMs) (5Server-i386)
- Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64)

Now I can sync the repos added from old content certs as well as from newly generated stage contents.

------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= Repository Synchronization Status =-

Last Refreshed: 16:30:15
(updated every 5 seconds, ctrl+c to exit)

Next Sync                    Last Sync                    Last Result         
------------------------------------------------------------------------------
Red Hat Enterprise Linux 6 Server - Optional from RHUI (Source RPMs) (6Server-x86_64)
In Progress                  Never                        Never       

Red Hat Update Infrastructure 1.2 (RPMs) (5Server-i386)
07-18-2011 21:53             07-18-2011 15:55             Success    

Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64)
07-18-2011 21:53             07-18-2011 15:55             Success    


                                  Connected: dhcp201-175.englab.pnq.redhat.com
------------------------------------------------------------------------------
^Crhui (sync) =>
Comment 12 wes hayutin 2011-08-01 21:40:48 UTC 
moving to release pending
Comment 13 wes hayutin 2012-05-31 12:57:45 UTC 
closing out, product released