Bug 711133

Summary: subscription-manager putting incorrect data in for sslclientkey in repo file
Product: Red Hat Enterprise Linux 6 Reporter: Tim Bielawa <tbielawa>
Component: subscription-managerAssignee: Bryan Kearney <bkearney>
Status: CLOSED ERRATA QA Contact: John Sefler <jsefler>
Severity: high Docs Contact:
Priority: urgent    
Version: 6.1CC: acarter, alikins, borgan, cduryee, ddumas, jwest, mkhusid
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
When the subscription-manager utility had been upgraded, it put incorrect data to the sslclientkey repository parameter value. Consequently, when the yum utility was executed to install a software, yum terminated with the "[Errno 14] problem with the local client certificate" error message. The bug in subscription-manager has been fixed and yum can now be run without any certificate errors.
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 17:15:07 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 712409    

Description Tim Bielawa 2011-06-06 16:03:42 UTC
Description of problem:

(This is using an employee subscription.)

After installing RHEL 6.1 software installation via yum breaks after upgrading subscription-manager. I believe the breakage is due to upgrading subscription-manager to a newer version (below).

After poking my nose around with bleanhar we figured out that this:

(Assuming we're using 123456 for the entitlement number)

The entitlement certificate is created:

/etc/pki/entitlement/123456.pem

The entitlement key is being created as:

/etc/pki/entitlement/key.pem

This seems to conflict with the yum repository configuration which expects these values for the client key and client cert:

sslclientkey = /etc/pki/entitlement/123456-key.pem
sslclientcert = /etc/pki/entitlement/123456.pem

Version-Release number of selected component (if applicable):

RHEL: 6.1

Post upgrading: subscription-manager: 0.95.14-1.el6_1.x86_64
Before upgrading: subscription-manager: 0.95.11-1.el6.x86_64.rpm


How reproducible:
For me, every RHEL 6.1 install this happen on.


Steps to Reproduce:
1. Download RHEL 6.1 boot iso from http://access.redhat.com
2. Boot VM or physical machine with iso
3. Use http://download.devel.redhat.com/released/RHEL-6-Server/6.1/x86_64/os/ for installation source.
4. Register with subscription-manager
4. Install some software
5. yum update
6. Attempt to install some more software


Actual results:
Yum error:

https://cdn.redhat.com/content....../package.rpm: [Errorno 14] problem with the local client certificate.


Expected results:
Software installs.


Additional info:
I could bring up another VM and see how the redhat.repo configuration looks before updating subscription-manager.


As a temporary work around you can make a symbolic link:

/etc/pki/entitlement/123456-key.pem -> /etc/pki/entitlement/key.pem

Also, I'm not sure why (due to my limited understanding of what files subscription-manager touches) cleaning and reregistering the system puts the configuration files into a working state.

Comment 2 Chris Duryee 2011-06-06 16:49:05 UTC
Where are you getting 0.95.14 from?

Comment 3 Tim Bielawa 2011-06-06 17:03:44 UTC
(In reply to comment #2)
> Where are you getting 0.95.14 from?

From the repository provided by subscription-manager. I believe this is http://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/

Comment 7 Chris Duryee 2011-06-06 20:01:27 UTC
Tim,

Based on our previous conversations, it sounds like you already did a re-register which is one way to fix the issue.

If you run into this again while we work on a fix, you can do the following:

mkdir /tmp/old-certs
mv /etc/pki/entitlement/* /tmp/old-certs
subscription-manager refresh

That should re-download the certs without requiring you to re-register.

Comment 8 Adrian Likins 2011-06-06 20:57:13 UTC
commit 909de27a3cfd3787199e96ae8ebc23c2140faeae
Author: Adrian Likins <alikins>
Date:   Mon Jun 6 15:17:52 2011 -0400

    711133: Handle updates from old style key.pem certs
    
    If we don't have a $SERIAL-key.pem associated with each
    cert file, consider that cert invalid, and try to update it.
    
    For the upgrade case, this means we write it with the new style
    Writer(), and get the new style $SERIAL-key.pem file format. So
    we can use this to migrate from old style certs to new style

Comment 9 Tim Bielawa 2011-06-07 13:03:48 UTC
If you have an RPM cut I'll be happy to test this out.

Comment 12 Adrian Likins 2011-06-07 16:27:17 UTC
Some test scenarios:

media has .11
0-day errata/current version is .14
I'll call the async errata version .15
"old style certs" == have /etc/pki/entitlements/key.pem instead of /etc/pki/entitlements/123123123-key.pem
"new style certs" == /etc/pki/entitlements/123123123-key.pem  /etc/pki/entitlements/123123123.pem



media (no updates) [register and subscribe with .11]:
- will get "old style" entitlment certs
 (for ex, /etc/pki/entitlement/12312312.pem and /etc/pki/entitlement/key.pem)
- This should work fine by itself

media install [register/subscribe with .11], then update to .14
- will have "old style" certs installed
- new repo definitions will have new style cert paths, aka:

sslclientkey = /etc/pki/entitlement/3444004732902194803-key.pem
sslclientcert = /etc/pki/entitlement/3444004732902194803.pem

- bug is that those paths are actually incorrect, because of the installed
old style paths [/etc/pki/entitlement/3444004732902194803.pem, /etc/pki/entitlement/key.pem]

This shows up as a yum failure

Workaround is to rm -f /etc/pki/entitlements/*

(forces certs to be refected and written out "new style". these should be
the same certs, just different filename for the key. No reregistering required)


media install [register/subscribe with .11], then update to .15 (async errata)
- will start with "old style certs"
- new code understands that not having the "new style" key file is invalid, and will update the certs and write them out as "new style"
- should be transparent to user 

media install, update to o-day (.14), [subscribe/register]
- should get new style certs installed, and new style certs in repo defs
- user shouldn't see any issues

media install, update to o-day (.14), [subscribe/register], update to async .15
- will already have new style certs/repos
- new code see certs are valid, doesn't mess with them
- user shouldn't see any issues

media install, update to async (.15), [subscribe/register]
- will get new style certs/repos
- user shouldn't see any issues

Comment 13 John Sefler 2011-06-08 21:45:42 UTC
The following test scenario is not working as expected with subscription-manager-0.95.14-1.git.1.909de27.el6.x86_64 / python-rhsm-0.95.6-1.el6.noarch...

media install [register/subscribe with .11], then update to .15 (async errata)
- will start with "old style certs"
- new code understands that not having the "new style" key file is invalid, and
will update the certs and write them out as "new style"
- should be transparent to user 

After the user upgrades from .11 to .15, the redhat.repo is getting blanked out despite the fact that the user is still registered and has the "old style" entitlements from .11 in /etc/pki/entitlement.  As a result yum repolist is blank.

moving bug back to NEW status...

Comment 15 Adrian Likins 2011-06-09 18:45:56 UTC
commit 74e4e5232b13fb24b3515abd89074b16c16f8421
Author: Adrian Likins <alikins>
Date:   Thu Jun 9 11:47:40 2011 -0400

    711133: new fix for old style to new style key format migrations
    
    If we see an old style key, go ahead and save it as the new format,
    instead of marking it invalid and waiting for CertLib.update to
    update it.
    
    This fixes an issue where the subscription-manager yum plugin would
    not update the keys, since it was never running CertLib.update

Comment 17 John Sefler 2011-06-10 14:22:23 UTC
Verifying the scenario:
 - a user has installed RHEL 6.1 from media (subscription-manager-0.95.11-1.el6_1/python-rhsm-0.95.6-1.el6.noarch.rpm is installed)
 - the user registers and subscribes to ONE subscription for RHEL (Note: subscribing to more than ONE subscription leads to bug 702398)
 - the user runs yum update which installs subscription-manager-0.95.14-1.el6_1
 - the user runs yum [install|repolist|...] and BANG! [Errno 14] problem with the local client certificate.

FIX:
 A) manual step
 # rm -f /etc/pki/entitlement/*
 # subscription-manager refresh

or

 B) manual step
 # subscription-manager identity       (Take note of the <identity-hash> returned)
 # subscription-manager register --username=<username> --password=<password> --consumerid=<identity-hash>

or

 C) manual step
 # subscription-manager unregister
 # subscription-manager register --username=<username> --password=<password> --autosubscribe

 - now the user can run yum [install|repolist|...] without any certificate errors (entitlement cert/key pairs are used)


Testing......................................

[root@jsefler-stage-6server tmp]# rpm -q subscription-manager python-rhsm
subscription-manager-0.95.11-1.el6.x86_64
python-rhsm-0.95.6-1.el6.noarch
[root@jsefler-stage-6server tmp]# subscription-manager register --username=qa
Password: 
53368b11-52e0-4c12-b30a-2fac08e58020 jsefler-stage-6server.usersys.redhat.com
[root@jsefler-stage-6server tmp]# subscription-manager  subscribe --auto
Installed Products:
   Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux Server - Not Installed
   Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux High Availability (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux Workstation - Not Installed
   Red Hat Enterprise Linux 6 Server - Not Subscribed
   Red Hat Enterprise Linux Load Balancer (for RHEL 6 Server) - Not Subscribed
   Red Hat Enterprise Linux Resilient Storage (for RHEL 6 Server) - Not Subscribed
   Red Hat Enterprise Linux High Availability (for RHEL 6 Server) - Not Subscribed
   Red Hat Enterprise Linux Scalable File System (for RHEL 6 Server) - Not Subscribed
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
2469656225150582984.pem  key.pem

^^^ Notice the "old style certs"/"pre bug 702398" present with subscription-manager-0.95.11-1

[root@jsefler-stage-6server tmp]# yum update subscription-manager python-rhsm
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
rhel-6-server-rpms                                                                                                          | 2.1 kB     00:00     
rhel-ha-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-lb-for-rhel-6-server-rpms                                                                                              | 2.4 kB     00:00     
rhel-rs-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-scalefs-for-rhel-6-server-rpms                                                                                         | 2.4 kB     00:00     
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package python-rhsm.noarch 0:0.95.6-1.el6 will be updated
---> Package python-rhsm.noarch 0:0.95.14-1.el6_1 will be an update
---> Package subscription-manager.x86_64 0:0.95.11-1.el6 will be updated
---> Package subscription-manager.x86_64 0:0.95.14-1.el6_1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================
 Package                                 Arch                      Version                             Repository                             Size
===================================================================================================================================================
Updating:
 python-rhsm                             noarch                    0.95.14-1.el6_1                     rhel-6-server-rpms                     33 k
 subscription-manager                    x86_64                    0.95.14-1.el6_1                     rhel-6-server-rpms                    292 k

Transaction Summary
===================================================================================================================================================
Upgrade       2 Package(s)

Total download size: 325 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): python-rhsm-0.95.14-1.el6_1.noarch.rpm                                                                               |  33 kB     00:00     
(2/2): subscription-manager-0.95.14-1.el6_1.x86_64.rpm                                                                      | 292 kB     00:00     
---------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                              167 kB/s | 325 kB     00:01     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : python-rhsm-0.95.14-1.el6_1.noarch                                                                                              1/4 
  Updating   : subscription-manager-0.95.14-1.el6_1.x86_64                                                                                     2/4 
  Cleanup    : subscription-manager-0.95.11-1.el6.x86_64                                                                                       3/4 
  Cleanup    : python-rhsm-0.95.6-1.el6.noarch                                                                                                 4/4 
rhel-ha-for-rhel-6-server-rpms/productid                                                                                    | 1.7 kB     00:00     
rhel-lb-for-rhel-6-server-rpms/productid                                                                                    | 1.7 kB     00:00     
rhel-rs-for-rhel-6-server-rpms/productid                                                                                    | 1.7 kB     00:00     
rhel-scalefs-for-rhel-6-server-rpms/productid                                                                               | 1.7 kB     00:00     
duration: 261(ms)
Installed products updated.

Updated:
  python-rhsm.noarch 0:0.95.14-1.el6_1                                subscription-manager.x86_64 0:0.95.14-1.el6_1                               

Complete!

[root@jsefler-stage-6server tmp]# yum install foo
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/highavailability/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/loadbalancer/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/resilientstorage/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/scalablefilesystem/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
Setting up Install Process
No package foo available.
Error: Nothing to do
[root@jsefler-stage-6server tmp]# 

^^^ BANG! Notice the [Errno 14] problem with the local client certificate

Now. let's apply the manual fix A)...

[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
2469656225150582984.pem  key.pem
[root@jsefler-stage-6server tmp]# rm -f /etc/pki/entitlement/*
[root@jsefler-stage-6server tmp]# subscription-manager refresh
All local data refreshed
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
2469656225150582984-key.pem  2469656225150582984.pem

^^^ Notice the "new style certs" have replaced the old style certs and yum install foo will no longer fail with [Errno 14] with subscription-manager-0.95.14-1

[root@jsefler-stage-6server tmp]# yum install foo
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
rhel-6-server-rpms                                                                                                          | 2.1 kB     00:00     
rhel-ha-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-lb-for-rhel-6-server-rpms                                                                                              | 2.4 kB     00:00     
rhel-rs-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-scalefs-for-rhel-6-server-rpms                                                                                         | 2.4 kB     00:00     
Setting up Install Process
No package foo available.
Error: Nothing to do

^^^ Manual fix A) works - no more [Errno 14] problem with the local client certificate


Now. let's backup to the bad state...

[root@jsefler-stage-6server tmp]# mv /etc/pki/entitlement/2469656225150582984-key.pem /etc/pki/entitlement/key.pem
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement
2469656225150582984.pem  key.pem
[root@jsefler-stage-6server tmp]# yum install foo
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/highavailability/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/loadbalancer/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/resilientstorage/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/scalablefilesystem/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
Setting up Install Process
No package foo available.
Error: Nothing to do

Now. let's apply the manual fix B)...

[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
2469656225150582984.pem  key.pem
[root@jsefler-stage-6server tmp]# subscription-manager unregister
System has been un-registered.
[root@jsefler-stage-6server tmp]# subscription-manager register --username=qa --autosubscribe
Password: 
9ecccef1-ecae-4627-a931-ccdc2989181f jsefler-stage-6server.usersys.redhat.com
Installed Products:
   Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux Server - Not Installed
   Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux High Availability (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux Workstation - Not Installed
   Red Hat Enterprise Linux 6 Server - Not Subscribed
   Red Hat Enterprise Linux Load Balancer (for RHEL 6 Server) - Not Subscribed
   Red Hat Enterprise Linux Resilient Storage (for RHEL 6 Server) - Not Subscribed
   Red Hat Enterprise Linux High Availability (for RHEL 6 Server) - Not Subscribed
   Red Hat Enterprise Linux Scalable File System (for RHEL 6 Server) - Not Subscribed
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
5422454651169117188-key.pem  5422454651169117188.pem

^^^ Notice the "new style certs" have replaced the old style certs and yum install foo will no longer fail with [Errno 14] with subscription-manager-0.95.14-1

[root@jsefler-stage-6server tmp]# yum install foo
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
rhel-6-server-rpms                                                                                                          | 2.1 kB     00:00     
rhel-ha-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-lb-for-rhel-6-server-rpms                                                                                              | 2.4 kB     00:00     
rhel-rs-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-scalefs-for-rhel-6-server-rpms                                                                                         | 2.4 kB     00:00     
Setting up Install Process
No package foo available.
Error: Nothing to do

^^^ Manual fix B) works - no more [Errno 14] problem with the local client certificate


Now. let's backup to the bad state...

[root@jsefler-stage-6server tmp]# mv /etc/pki/entitlement/2469656225150582984-key.pem /etc/pki/entitlement/key.pem
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement
2469656225150582984.pem  key.pem
[root@jsefler-stage-6server tmp]# yum install foo
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/highavailability/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/loadbalancer/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/resilientstorage/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/scalablefilesystem/os/repodata/repomd.xml: [Errno 14] problem with the local client certificate
Trying other mirror.
Setting up Install Process
No package foo available.
Error: Nothing to do

Now. let's apply the manual fix C)...

[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
2469656225150582984.pem  key.pem
[root@jsefler-stage-6server tmp]# subscription-manager identity
Current identity is: 9662ae86-5b81-4c7a-a6ba-218d2f7271e1 name: jsefler-stage-6server.usersys.redhat.com
[root@jsefler-stage-6server tmp]# subscription-manager clean
All local data removed
[root@jsefler-stage-6server tmp]# subscription-manager register --consumerid=9662ae86-5b81-4c7a-a6ba-218d2f7271e1 --username=qa
Password: 
9662ae86-5b81-4c7a-a6ba-218d2f7271e1 jsefler-stage-6server.usersys.redhat.com
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
1690843792521143389-key.pem  1690843792521143389.pem

^^^ Notice the "new style certs" have replaced the old style certs and yum install foo will no longer fail with [Errno 14] with subscription-manager-0.95.14-1

[root@jsefler-stage-6server tmp]# yum install foo
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
rhel-6-server-rpms                                                                                                          | 2.1 kB     00:00     
rhel-ha-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-lb-for-rhel-6-server-rpms                                                                                              | 2.4 kB     00:00     
rhel-rs-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-scalefs-for-rhel-6-server-rpms                                                                                         | 2.4 kB     00:00     
Setting up Install Process
No package foo available.
Error: Nothing to do

^^^ Manual fix C) works - no more [Errno 14] problem with the local client certificate



NOTE: I believe we need a technical note based on this comment since one of the manual (A, B, or C) intervention steps outlined above is needed.  Also note that this technical note only applies to a user who has deviated from the default software updates model (RHN Classic) to use subscription-manager and is experiencing a yum "[Errno 14] problem with the local client certificate."

Comment 18 John Sefler 2011-06-10 14:23:24 UTC
Verifying the scenario:
 - a user has installed RHEL 6.1 from media (subscription-manager-0.95.11-1.el6_1 is installed)
 - the user registers and subscribes to ONE subscription for RHEL (Note: subscribing to more than ONE subscription leads to bug 702398)
 - the user runs yum update which installs subscription-manager-0.95.15-1.el6_1
 - the user runs yum [install|repolist|...] and the migration to the certificate keys is transparent to the user ("new style" entitlement cert/key pairs are used).


Testing......................................

[root@jsefler-stage-6server tmp]# rpm -q subscription-manager python-rhsm
subscription-manager-0.95.11-1.el6.x86_64
python-rhsm-0.95.6-1.el6.noarch
[root@jsefler-stage-6server tmp]# subscription-manager register --username=qa --autosubscribe
Password: 
02df9f79-1459-43e1-8129-cabb66ff821f jsefler-stage-6server.usersys.redhat.com
Installed Products:
   Red Hat Enterprise Linux Server - Not Installed
   Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux Workstation - Not Installed
   Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux High Availability (for RHEL Server) - Not Installed
   Red Hat Enterprise Linux 6 Server - Not Subscribed
   Red Hat Enterprise Linux Load Balancer (for RHEL 6 Server) - Not Subscribed
   Red Hat Enterprise Linux Resilient Storage (for RHEL 6 Server) - Not Subscribed
   Red Hat Enterprise Linux High Availability (for RHEL 6 Server) - Not Subscribed
   Red Hat Enterprise Linux Scalable File System (for RHEL 6 Server) - Not Subscribed
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
2985483936298596146.pem  key.pem
[root@jsefler-stage-6server tmp]# 


^^^ Notice the "old style certs"/"pre bug 702398" present with subscription-manager-0.95.11-1

Now let's yum update to subscription-manager-0.95.15-1...
NOTE: IN THE FOLLOWING STEPS I USE YUM LOCAL INSTALL TO MANUALLY UPDATE TO A PRE-RELEASESED DEVELOPMENT BUILD OF subscription-manager-0.95.15-1 SINCE THE ERRATA HAS NOT YET BEEN COMPOSED

[root@jsefler-stage-6server tmp]# yum localinstall --nogpgcheck subscription-manager.x86_64.rpm python-rhsm-0.95.14-1.el6_1.noarch.rpm
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
Setting up Local Package Process
Examining subscription-manager.x86_64.rpm: subscription-manager-0.95.15-1.git.0.a616959.el6.x86_64
Marking subscription-manager.x86_64.rpm as an update to subscription-manager-0.95.11-1.el6.x86_64
rhel-6-server-rpms                                                                                                          | 2.1 kB     00:00     
rhel-ha-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-lb-for-rhel-6-server-rpms                                                                                              | 2.4 kB     00:00     
rhel-rs-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-scalefs-for-rhel-6-server-rpms                                                                                         | 2.4 kB     00:00     
Examining python-rhsm-0.95.14-1.el6_1.noarch.rpm: python-rhsm-0.95.14-1.el6_1.noarch
Marking python-rhsm-0.95.14-1.el6_1.noarch.rpm as an update to python-rhsm-0.95.6-1.el6.noarch
Resolving Dependencies
--> Running transaction check
---> Package python-rhsm.noarch 0:0.95.6-1.el6 will be updated
---> Package python-rhsm.noarch 0:0.95.14-1.el6_1 will be an update
---> Package subscription-manager.x86_64 0:0.95.11-1.el6 will be updated
---> Package subscription-manager.x86_64 0:0.95.15-1.git.0.a616959.el6 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================
 Package                          Arch               Version                                 Repository                                       Size
===================================================================================================================================================
Updating:
 python-rhsm                      noarch             0.95.14-1.el6_1                         /python-rhsm-0.95.14-1.el6_1.noarch             109 k
 subscription-manager             x86_64             0.95.15-1.git.0.a616959.el6             /subscription-manager.x86_64                    1.2 M

Transaction Summary
===================================================================================================================================================
Upgrade       2 Package(s)

Total size: 1.3 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : python-rhsm-0.95.14-1.el6_1.noarch                                                                                              1/4 
  Updating   : subscription-manager-0.95.15-1.git.0.a616959.el6.x86_64                                                                         2/4 
  Cleanup    : subscription-manager-0.95.11-1.el6.x86_64                                                                                       3/4 
  Cleanup    : python-rhsm-0.95.6-1.el6.noarch                                                                                                 4/4 
duration: 244(ms)
Installed products updated.

Updated:
  python-rhsm.noarch 0:0.95.14-1.el6_1                          subscription-manager.x86_64 0:0.95.15-1.git.0.a616959.el6                         

Complete!
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
2985483936298596146.pem  key.pem

^^^ Notice that the "old style" certs/key are still in place.
Now let's run yum repolist and we should NOT get "[Errno 14] problem with the local client certificate" and the certs should get converted to the "new style"...

[root@jsefler-stage-6server tmp]# yum repolist
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
rhel-6-server-rpms                                                                                                          | 2.1 kB     00:00     
rhel-ha-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-lb-for-rhel-6-server-rpms                                                                                              | 2.4 kB     00:00     
rhel-rs-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-scalefs-for-rhel-6-server-rpms                                                                                         | 2.4 kB     00:00     
repo id                                             repo name                                                                                status
rhel-6-server-rpms                                  Red Hat Enterprise Linux 6 Server (RPMs)                                                 5,021
rhel-ha-for-rhel-6-server-rpms                      Red Hat Enterprise Linux High Availability (for RHEL 6 Server) (RPMs)                       87
rhel-lb-for-rhel-6-server-rpms                      Red Hat Enterprise Linux Load Balancer (for RHEL 6 Server) (RPMs)                            2
rhel-rs-for-rhel-6-server-rpms                      Red Hat Enterprise Linux Resilient Storage (for RHEL 6 Server) (RPMs)                      100
rhel-scalefs-for-rhel-6-server-rpms                 Red Hat Enterprise Linux Scalable File System (for RHEL 6 Server) (RPMs)                     7
repolist: 5,217
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
2985483936298596146-key.pem  2985483936298596146.pem  key.pem
[root@jsefler-stage-6server tmp]# 

^^^ VERIFIED: No "[Errno 14] problem with the local client certificate" and the "new style" cert/key pairing is present.  Note the old key.pem was left behind.  Consider it abandoned.





I would also like to verify one more scenario (when the user has subscribed to multiple subscriptions using subscription-manager-0.95.11-1 and has somehow (probably using RHN Classic) upgraded to subscription-manager-0.95.15-1).
Let's make sure that ALL of the "old style" entitlement certs get converted to the "new style" cert/key pairs upon calling yum...
[root@jsefler-stage-6server tmp]# rpm -q subscription-manager python-rhsm
subscription-manager-0.95.11-1.el6.x86_64
python-rhsm-0.95.6-1.el6.noarch
[root@jsefler-stage-6server tmp]# subscription-manager register --username=qa
Password: 
01c13166-a52b-40cb-8814-2d478aa581fc jsefler-stage-6server.usersys.redhat.com
[root@jsefler-stage-6server tmp]# subscription-manager list --avail | grep PoolId
PoolId:            	8a85f9812ede00af012edf01c8965ceb
PoolId:            	8a85f9812ede00af012edf01c89f5cf9
PoolId:            	8a85f9812ede00af012edf01c8a65d04
PoolId:            	8a85f981302cbaf2013046b66d9c761a
PoolId:            	8a85f981302cbaf2013046b7cf077694
PoolId:            	8a85f981302cbaf2013046bb01bb7699
PoolId:            	8a85f981302cbaf20130475bf7f01895
PoolId:            	8a85f981302cbaf20130475bf8231897
PoolId:            	8a85f981302cbaf201304761614a1b76
PoolId:            	8a85f981302cbaf201304b4df59206fe
PoolId:            	8a85f981302cbaf201304b589d620720
PoolId:            	8a85f981302cbaf201304b7440e1073f
PoolId:            	8a85f981302cbaf201304b7a341c0767
[root@jsefler-stage-6server tmp]# subscription-manager subscribe --pool=8a85f9812ede00af012edf01c8965ceb --pool=8a85f9812ede00af012edf01c89f5cf9 --pool=8a85f9812ede00af012edf01c8a65d04 --pool=8a85f981302cbaf2013046b66d9c761a --pool=8a85f981302cbaf2013046b7cf077694 --pool=8a85f981302cbaf2013046bb01bb7699 --pool=8a85f981302cbaf20130475bf7f01895 --pool=8a85f981302cbaf20130475bf8231897 --pool=8a85f981302cbaf201304761614a1b76 --pool=8a85f981302cbaf201304b4df59206fe --pool=8a85f981302cbaf201304b589d620720 --pool=8a85f981302cbaf201304b7440e1073f --pool=8a85f981302cbaf201304b7a341c0767
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
1246646451713520525.pem  2371247288929630499.pem  4236107324522706759.pem  8341798893048093573.pem  9187337311637326907.pem
1399677072389736269.pem  3145343669413397581.pem  4767853026639665791.pem  8423304058618120021.pem  key.pem
1866071580938659597.pem  3591687347680153604.pem  4784051234343593943.pem  891865983006711628.pem
[root@jsefler-stage-6server tmp]# yum localinstall --nogpgcheck subscription-manager.x86_64.rpm python-rhsm-0.95.14-1.el6_1.noarch.rpm
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
Setting up Local Package Process
Examining subscription-manager.x86_64.rpm: subscription-manager-0.95.15-1.git.0.a616959.el6.x86_64
Marking subscription-manager.x86_64.rpm as an update to subscription-manager-0.95.11-1.el6.x86_64
rhel-6-server-rpms                                                                                                          | 2.1 kB     00:00     
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/highavailability/os/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "NSS: private key not found for certificate: PEM Token #1:8341798893048093573.pem"
Trying other mirror.
rhel-lb-for-rhel-6-server-rpms                                                                                              | 2.4 kB     00:00     
rhel-rs-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-scalefs-for-rhel-6-server-rpms                                                                                         | 2.4 kB     00:00     
Examining python-rhsm-0.95.14-1.el6_1.noarch.rpm: python-rhsm-0.95.14-1.el6_1.noarch
Marking python-rhsm-0.95.14-1.el6_1.noarch.rpm as an update to python-rhsm-0.95.6-1.el6.noarch
Resolving Dependencies
--> Running transaction check
---> Package python-rhsm.noarch 0:0.95.6-1.el6 will be updated
---> Package python-rhsm.noarch 0:0.95.14-1.el6_1 will be an update
---> Package subscription-manager.x86_64 0:0.95.11-1.el6 will be updated
---> Package subscription-manager.x86_64 0:0.95.15-1.git.0.a616959.el6 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================
 Package                          Arch               Version                                 Repository                                       Size
===================================================================================================================================================
Updating:
 python-rhsm                      noarch             0.95.14-1.el6_1                         /python-rhsm-0.95.14-1.el6_1.noarch             109 k
 subscription-manager             x86_64             0.95.15-1.git.0.a616959.el6             /subscription-manager.x86_64                    1.2 M

Transaction Summary
===================================================================================================================================================
Upgrade       2 Package(s)

Total size: 1.3 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : python-rhsm-0.95.14-1.el6_1.noarch                                                                                              1/4 
  Updating   : subscription-manager-0.95.15-1.git.0.a616959.el6.x86_64                                                                         2/4 
  Cleanup    : subscription-manager-0.95.11-1.el6.x86_64                                                                                       3/4 
  Cleanup    : python-rhsm-0.95.6-1.el6.noarch                                                                                                 4/4 
duration: 324(ms)
Installed products updated.

Updated:
  python-rhsm.noarch 0:0.95.14-1.el6_1                          subscription-manager.x86_64 0:0.95.15-1.git.0.a616959.el6                         

Complete!
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
1246646451713520525.pem  2371247288929630499.pem  4236107324522706759.pem  8341798893048093573.pem  9187337311637326907.pem
1399677072389736269.pem  3145343669413397581.pem  4767853026639665791.pem  8423304058618120021.pem  key.pem
1866071580938659597.pem  3591687347680153604.pem  4784051234343593943.pem  891865983006711628.pem
[root@jsefler-stage-6server tmp]# yum repolist
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
rhel-6-server-rpms                                                                                                          | 2.1 kB     00:00     
rhel-ha-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-lb-for-rhel-6-server-rpms                                                                                              | 2.4 kB     00:00     
rhel-rs-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-scalefs-for-rhel-6-server-rpms                                                                                         | 2.4 kB     00:00     
repo id                                             repo name                                                                                status
rhel-6-server-rpms                                  Red Hat Enterprise Linux 6 Server (RPMs)                                                 5,021
rhel-ha-for-rhel-6-server-rpms                      Red Hat Enterprise Linux High Availability (for RHEL 6 Server) (RPMs)                       87
rhel-lb-for-rhel-6-server-rpms                      Red Hat Enterprise Linux Load Balancer (for RHEL 6 Server) (RPMs)                            2
rhel-rs-for-rhel-6-server-rpms                      Red Hat Enterprise Linux Resilient Storage (for RHEL 6 Server) (RPMs)                      100
rhel-scalefs-for-rhel-6-server-rpms                 Red Hat Enterprise Linux Scalable File System (for RHEL 6 Server) (RPMs)                     7
repolist: 5,217
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
1246646451713520525-key.pem  2371247288929630499-key.pem  4236107324522706759-key.pem  8341798893048093573-key.pem  9187337311637326907-key.pem
1246646451713520525.pem      2371247288929630499.pem      4236107324522706759.pem      8341798893048093573.pem      9187337311637326907.pem
1399677072389736269-key.pem  3145343669413397581-key.pem  4767853026639665791-key.pem  8423304058618120021-key.pem  key.pem
1399677072389736269.pem      3145343669413397581.pem      4767853026639665791.pem      8423304058618120021.pem
1866071580938659597-key.pem  3591687347680153604-key.pem  4784051234343593943-key.pem  891865983006711628-key.pem
1866071580938659597.pem      3591687347680153604.pem      4784051234343593943.pem      891865983006711628.pem
[root@jsefler-stage-6server tmp]# 

^^^  VERIFIED that even when multiple entitlement exist, the fix in subscription-manager-0.95.15-1 correctly migrates the "old style" certs to the "new style" cert/key pairs (with the key.pem left behind as abandoned)

[root@jsefler-stage-6server tmp]# yum repolist
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
rhel-6-server-rpms                                                                                                          | 2.1 kB     00:00     
rhel-ha-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-lb-for-rhel-6-server-rpms                                                                                              | 2.4 kB     00:00     
rhel-rs-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-scalefs-for-rhel-6-server-rpms                                                                                         | 2.4 kB     00:00     
repo id                                             repo name                                                                                status
rhel-6-server-rpms                                  Red Hat Enterprise Linux 6 Server (RPMs)                                                 5,021
rhel-ha-for-rhel-6-server-rpms                      Red Hat Enterprise Linux High Availability (for RHEL 6 Server) (RPMs)                       87
rhel-lb-for-rhel-6-server-rpms                      Red Hat Enterprise Linux Load Balancer (for RHEL 6 Server) (RPMs)                            2
rhel-rs-for-rhel-6-server-rpms                      Red Hat Enterprise Linux Resilient Storage (for RHEL 6 Server) (RPMs)                      100
rhel-scalefs-for-rhel-6-server-rpms                 Red Hat Enterprise Linux Scalable File System (for RHEL 6 Server) (RPMs)                     7
repolist: 5,217
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
1246646451713520525-key.pem  2371247288929630499-key.pem  4236107324522706759-key.pem  8341798893048093573-key.pem  9187337311637326907-key.pem
1246646451713520525.pem      2371247288929630499.pem      4236107324522706759.pem      8341798893048093573.pem      9187337311637326907.pem
1399677072389736269-key.pem  3145343669413397581-key.pem  4767853026639665791-key.pem  8423304058618120021-key.pem  key.pem
1399677072389736269.pem      3145343669413397581.pem      4767853026639665791.pem      8423304058618120021.pem
1866071580938659597-key.pem  3591687347680153604-key.pem  4784051234343593943-key.pem  891865983006711628-key.pem
1866071580938659597.pem      3591687347680153604.pem      4784051234343593943.pem      891865983006711628.pem

^^^  VERIFIED that a second call to yum repolist does not alter the migrated entitlement cert/key pairs

[root@jsefler-stage-6server tmp]# rm /etc/pki/entitlement/key.pem 
rm: remove regular file `/etc/pki/entitlement/key.pem'? y
[root@jsefler-stage-6server tmp]# yum repolist
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
rhel-6-server-rpms                                                                                                          | 2.1 kB     00:00     
rhel-ha-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-lb-for-rhel-6-server-rpms                                                                                              | 2.4 kB     00:00     
rhel-rs-for-rhel-6-server-rpms                                                                                              | 2.2 kB     00:00     
rhel-scalefs-for-rhel-6-server-rpms                                                                                         | 2.4 kB     00:00     
repo id                                             repo name                                                                                status
rhel-6-server-rpms                                  Red Hat Enterprise Linux 6 Server (RPMs)                                                 5,021
rhel-ha-for-rhel-6-server-rpms                      Red Hat Enterprise Linux High Availability (for RHEL 6 Server) (RPMs)                       87
rhel-lb-for-rhel-6-server-rpms                      Red Hat Enterprise Linux Load Balancer (for RHEL 6 Server) (RPMs)                            2
rhel-rs-for-rhel-6-server-rpms                      Red Hat Enterprise Linux Resilient Storage (for RHEL 6 Server) (RPMs)                      100
rhel-scalefs-for-rhel-6-server-rpms                 Red Hat Enterprise Linux Scalable File System (for RHEL 6 Server) (RPMs)                     7
repolist: 5,217
[root@jsefler-stage-6server tmp]# ls /etc/pki/entitlement/
1246646451713520525-key.pem  2371247288929630499-key.pem  4236107324522706759-key.pem  8341798893048093573-key.pem  9187337311637326907-key.pem
1246646451713520525.pem      2371247288929630499.pem      4236107324522706759.pem      8341798893048093573.pem      9187337311637326907.pem
1399677072389736269-key.pem  3145343669413397581-key.pem  4767853026639665791-key.pem  8423304058618120021-key.pem
1399677072389736269.pem      3145343669413397581.pem      4767853026639665791.pem      8423304058618120021.pem
1866071580938659597-key.pem  3591687347680153604-key.pem  4784051234343593943-key.pem  891865983006711628-key.pem
1866071580938659597.pem      3591687347680153604.pem      4784051234343593943.pem      891865983006711628.pem
[root@jsefler-stage-6server tmp]# 


^^^ VERIFIED that manual removal of the abandoned key.pem does not harm the "new style" entitlement cert/key pairs upon calling yum repolist

Comment 19 Tomas Capek 2011-06-24 08:56:23 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
When the subscription-manager utility had been upgraded, it put incorrect data to the sslclientkey repository parameter value. Consequently, when the yum utility was executed to install a software, yum terminated with the "[Errno 14] problem with the local client certificate" error message. The bug in subscription-manager has been fixed and yum can now be run without any certificate errors.

Comment 20 Tim Bielawa 2011-07-27 13:25:21 UTC
I just wanted to report that I can give this fix my personal +1. I provisioned a new VM and the install + first upgrade process was flawless.

Thanks for your timely response on this issue!

Comment 21 errata-xmlrpc 2011-12-06 17:15:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1695.html