Bug 827530

Summary: consumer identity cert goes invalid months before its Validity end date
Product: Red Hat Enterprise Linux 6 Reporter: John Sefler <jsefler>
Component: subscription-managerAssignee: Devan Goodwin <dgoodwin>
Status: CLOSED NOTABUG QA Contact: Entitlement Bugs <entitlement-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.3CC: dgoodwin, khong
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-06 15:19:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 738066    

Description John Sefler 2012-06-01 17:40:50 UTC
Description of problem:
After registering to stage and or production, the consumer cert given is valid for one year.  However after advancing the system clock ahead less than a year, we get a "certificate verify failed"

Version-Release number of selected component (if applicable):
[root@jsefler-63server ~]# rpm -q python-rhsm subscription-manager
python-rhsm-0.99.12-1.el6.noarch
subscription-manager-0.99.19-1.el6.x86_64

[root@jsefler-63server ~]# rpm -q m2crypto
m2crypto-0.20.2-7.el6.x86_64


How reproducible:


Steps to Reproduce:
[root@jsefler-63server ~]# subscription-manager config --server.hostname=subscription.rhn.stage.redhat.com
[root@jsefler-63server ~]# subscription-manager register --username stage_test_12
Password: 
The system has been registered with id: 7813c8df-d23b-481c-a93e-fee1accbff32 
[root@jsefler-63server ~]# openssl x509 -text -in /etc/pki/consumer/cert.pem | grep Validity -A2
        Validity
            Not Before: Jun  1 17:25:53 2012 GMT
            Not After : Jun  1 17:25:53 2013 GMT
[root@jsefler-63server ~]# date 010100002013
Tue Jan  1 00:00:00 EST 2013
[root@jsefler-63server ~]# subscription-manager identity
Current identity is: 7813c8df-d23b-481c-a93e-fee1accbff32
name: jsefler-63server.usersys.redhat.com
org name: Anonymous User
org id: 8a99f981370984da0137098639790022
[root@jsefler-63server ~]# date 020100002013
Fri Feb  1 00:00:00 EST 2013
[root@jsefler-63server ~]# subscription-manager identity
certificate verify failed



Actual results:
Advancing the system clock 7 months ahead is ok
Advancing the system clock 8 months ahead fails

Expected results:
The consumer cert should remain valid until the end of its Validity date.

Additional info:

Comment 1 John Sefler 2012-06-01 17:43:44 UTC
Also failed with m2crypto version...

[root@jsefler-63server ~]# rpm -q m2crypto
m2crypto-0.20.2-9.el6.x86_64

Comment 2 John Sefler 2012-06-02 13:34:04 UTC
Also note that identity --regenerate will not work when in this state...

[root@jsefler-63server ~]# subscription-manager identity --regenerate
certificate verify failed

Comment 6 Devan Goodwin 2012-06-05 15:03:37 UTC
Reproduced with a modified local Candlepin, set to give just 2 months for the identity cert, regenerate my systems certificate, and I am immediately unable to verify my certificate. (no system date manipulation at all)

(root@redhat ~) $ subscription-manager identity                                                      
certificate verify failed
(root@redhat ~) $ openssl x509 -text -in /etc/pki/consumer/cert.pem| grep -A2 Validity
        Validity
            Not Before: Jun  5 15:01:33 2012 GMT
            Not After : Aug  5 15:01:33 2012 GMT
(root@redhat ~) $ date
Tue Jun  5 12:03:08 ADT 2012
(root@redhat ~) $

Comment 7 Devan Goodwin 2012-06-05 15:41:16 UTC
Good news, I think this may not be a bug after all. The reason is because the server's CA certificate we use to verify the signatures is expired. You can see the same error even if you go straight to openssl.

$ openssl verify -CAfile /etc/rhsm/ca/candlepin-ca.pem /etc/pki/consumer/cert.pem                  
/etc/pki/consumer/cert.pem: CN = localhost, C = US, L = Raleigh
error 10 at 1 depth lookup:certificate has expired
OK

This prompted me to look at my candlepin-ca.pem (which was very very old on my dev machine, I often use insecure=1 in rhsm.conf):

$ openssl x509 -text -in /etc/rhsm/ca/candlepin-ca.pem| grep -A2 Validity                          
        Validity
            Not Before: Apr  1 18:57:50 2011 GMT
            Not After : Mar 31 18:57:50 2012 GMT

So my CA cert was expired for a couple months now. 

I regenerated my server certificate, and did the following:

(root@redhat ~) $ cp /etc/candlepin/certs/candlepin-ca.crt /etc/rhsm/ca/candlepin-ca.pem
(root@redhat ~) $ subscription-manager clean                                                         
All local data removed
(root@redhat ~) $ subscription-manager register --username=admin --password=admin --force --org=admin
The system has been registered with id: 447ee2c3-6284-472f-84ac-bc68a980cd78 
(root@redhat ~) $ env PYTHONPATH=/home/dgoodwin/src/subscription-manager/src /home/dgoodwin/src/subscription-manage
r/src/subscription-manager identity
Current identity is: 447ee2c3-6284-472f-84ac-bc68a980cd78
name: redhat.local.rm-rf.ca
org name: Admin Owner
org id: ff80808137bd1eef0137bd1f06390009
(root@redhat ~) $ openssl x509 -text -in /etc/rhsm/ca/candlepin-ca.pem| grep -A2 Validity
        Validity
            Not Before: Jun  5 15:33:45 2012 GMT
            Not After : Jun  5 15:33:45 2013 GMT
(root@redhat ~) $ openssl verify -CAfile /etc/rhsm/ca/candlepin-ca.pem /etc/pki/consumer/cert.pem
/etc/pki/consumer/cert.pem: OK

Both m2crypto and openssl now think the certificate is valid.

Comment 8 Devan Goodwin 2012-06-05 16:12:59 UTC
Verified that both Katello and Headpin are generating this certificate for 25 years, so this is really only an issue for developer deployments (I will adjust this in deploy script and cpsetup today) and for QE when pushing the system date way into the future.

Comment 9 Keqin Hong 2012-06-06 06:17:30 UTC
Hi Devan

By looking at Comment 6 - Comment 8, I don't quite understand that it is the same issue as reported in Comment 0.
 
Notice that in comment 0, that the bug was reported against stage or prod candlepin.

> [root@jsefler-63server ~]# subscription-manager config
> --server.hostname=subscription.rhn.stage.redhat.com

I checked that both candlepin-stage and redhat-uep.pem are valid through 2030.

# openssl x509 -text -in /etc/rhsm/ca/candlepin-stage.pem | grep Validity -A2
        Validity
            Not Before: Oct 26 20:12:21 2010 GMT
            Not After : Oct 21 20:12:21 2030 GMT

# openssl x509 -text -in /etc/rhsm/ca/redhat-uep.pem | grep Validity -A2
        Validity
            Not Before: Oct  4 13:27:48 2010 GMT
            Not After : Sep 29 13:27:48 2030 GMT

Could you explain it a bit more?

Comment 10 Devan Goodwin 2012-06-06 12:28:15 UTC
Very good catch Keqin, my apologies, I had a reproducer, but mistakenly assumed it was the same issue. It does not appear to be. Re-opening.

Comment 11 Devan Goodwin 2012-06-06 15:19:08 UTC
Ok so we have traced down this issue as well, the result is quite similar to the previous reason.

redhat-uep.pem is not used to verify the identity certificates, actually we trust them but instead verify the certificate the server returns during the SSL handshake. This can be extracted from the output of: 

openssl s_client -connect subscription.rhn.redhat.com:443

Save the certificate portion into a separate file, and view with:

openssl x509 -text -in servercert.pem

In here you will see the server cert's expiry date:

        Validity
            Not Before: Feb  4 06:19:59 2011 GMT
            Not After : Feb  3 06:19:59 2013 GMT

So any time we bump the client system date beyond Feb 3 2013, the certificate validation will start to fail because the server's certificate has expired.

I have been informed this is ok and normal, the server cert is regenerated yearly and this does not require any client changes.

You can see the results:

(root@rhel6 /etc/rhsm/ca) $ date 020100002013
Fri Feb  1 00:00:00 AST 2013
(root@rhel6 /etc/rhsm/ca) $ subscription-manager identity
Current identity is: 128b7f48-a1e5-41d0-b94c-25c0dc3869bd
name: rhel6.local.rm-rf.ca
org name: 5894300
org id: 8a85f9812f035407012f273805c3284a
(root@rhel6 /etc/rhsm/ca) $ date 020400002013            
Mon Feb  4 00:00:00 AST 2013
(root@rhel6 /etc/rhsm/ca) $ subscription-manager identity
certificate verify failed

In short everything is behaving ok, it's just caused by pushing the system date beyond the server's date, which causes validation to fail.

Re-closing.