| Summary: | CDS sync fails with the below issue | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Update Infrastructure for Cloud Providers | Reporter: | Kedar Bidarkar <kbidarka> | ||||||
| Component: | CDS | Assignee: | James Slagle <jslagle> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | wes hayutin <whayutin> | ||||||
| Severity: | unspecified | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 2.0.2 | CC: | jslagle, kbidarka, sghai | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2012-03-12 19:40:19 UTC | Type: | --- | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Attachments: |
|
||||||||
|
Description
Kedar Bidarkar
2012-02-08 10:08:26 UTC
Created attachment 560209 [details]
cds sync issue
/var/log/ssl_error_log
Please let us know if there are any change in the steps. Currently we are not using ca-chain certs for testing. Issue was caused by bad merge in repo_cert_utils.py after rebasing on newer pulp. Fixed in commit 7dba257bcc16c40b72f28d6d1a2a95783907fb0c to pulp rhui branch. Built into new pulp packages at: https://brewweb.devel.redhat.com/buildinfo?buildID=198024 I'll ask rel-eng to spin a new ISO tomorrow. looks fixed in the brew rpms.. wait for brew iso spin to verify Built in latest QA iso: http://download.lab.bos.redhat.com/devel/candidates/RHEL-6.2-RHUI-2.0.2-20120209.0/2.0.2/Server/x86_64/iso/ Tested with latest rhui2.0.2 iso ( RHEL-6.2-RHUI-2.0.2-20120209.0-Server-x86_64-DVD1.iso ).
CDS sync is working fine, I synced two CDS nodes with rhel6 repos.
--
------------------------------------------------------------------------------
rhui (cds) => l
-= RHUI Content Delivery Servers =-
cds2
cds1
------------------------------------------------------------------------------
--
------------------------------------------------------------------------------
-= Red Hat Update Infrastructure Management Tool =-
-= CDS Synchronization Status =-
Last Refreshed: 06:01:19
(updated every 5 seconds, ctrl+c to exit)
cds1 ........................................................ [ UP ]
cds2 ........................................................ [ UP ]
Next Sync Last Sync Last Result
------------------------------------------------------------------------------
cds1
02-10-2012 09:28 02-10-2012 04:36 Success
cds2
02-10-2012 09:29 02-10-2012 04:36 Success
Connected: ip-10-124-57-219.ec2.internal
------------------------------------------------------------------------------
------------------------------------------------------------------------------
-= Red Hat Update Infrastructure Management Tool =-
-= CDS Synchronization Status =-
Last Refreshed: 01:33:25
(updated every 5 seconds, ctrl+c to exit)
CDS1_173_191 ................................................ [ UP ]
CDS2_109_224 ................................................ [ UP ]
Next Sync Last Sync Last Result
------------------------------------------------------------------------------
CDS1_173_191
02-14-2012 03:10 02-13-2012 21:12 Success
CDS2_109_224
02-14-2012 03:10 02-13-2012 21:12 Success
Connected: ip-10-80-202-84.ec2.internal
------------------------------------------------------------------------------
After expiration of entitlement ca cert and identity cert, I planned to add other valid certs for both and then the cds sync fails. I unregistered cds nodes and re-registered them again and still it fails. from /var/log/httpd/ssl_error_log [Wed Feb 15 11:33:41 2012] [error] [client 10.224.1.234] Using a CRL Stack with 0 CRL(s)Client certificate did not match the repo consumer CA certificate [Wed Feb 15 11:33:41 2012] [error] [client 10.224.1.234] user /CN=Red Hat Update Infrastructure: authentication failure for "/pulp/repos/content/dist/rhel/rhui/server/6/6Server/x86_64/os/repodata/repomd.xml": Password Mismatch [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Cert verification failed against 1 ca cert(s) and 0 CRL(s) [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Current Time: <Wed Feb 15 11:33:41 2012> [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Certificate: [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Data: [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Version: 3 (0x2) [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Serial Number: 259 (0x103) [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Signature Algorithm: sha1WithRSAEncryption [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Issuer: C=IN, ST=MH, L=Pune, O=RH, OU=Engg, CN=RHUI Entitlement CA [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Validity [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Not Before: Feb 15 13:28:14 2012 GMT [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Not After : Feb 14 13:28:14 2013 GMT [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Subject: CN=Red Hat Update Infrastructure [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Subject Public Key Info: [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Public Key Algorithm: rsaEncryption --------------------------------------------------- [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Certificate to verify: [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] \tsubject=</CN=Red Hat Update Infrastructure>, issuer=</C=IN/ST=MH/L=Pune/O=RH/OU=Engg/CN=RHUI Entitlement CA>, subject.as_hash=<2273005625>, issuer.as_hash=<1839428531>, fingerprint=<74DC25257EA7EECB53EF199BA8891C7E>, serial=<259>, version=<2>, check_ca=<0>, notBefore=<Feb 15 13:28:14 2012 GMT>, notAfter=<Feb 14 13:28:14 2013 GMT> [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Using a CA Chain with 1 cert(s) [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] \tCA: subject=</C=IN/ST=MH/L=Pune/O=RH/OU=Engg/CN=RHUI Entitlement CA>, issuer=</C=IN/ST=MH/L=Pune/O=RH/OU=Engg/CN=RHUI Entitlement CA>, subject.as_hash=<1839428531>, issuer.as_hash=<1839428531>, fingerprint=<BBB906AD4BE2E7949169E9AEED1C23FF>, serial=<14155745042073364165>, version=<2>, check_ca=<1>, notBefore=<Feb 14 08:07:31 2012 GMT>, notAfter=<Feb 15 08:07:31 2012 GMT> [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] Using a CRL Stack with 0 CRL(s)Client certificate did not match the repo consumer CA certificate [Wed Feb 15 11:33:41 2012] [error] [client 10.100.207.211] user /CN=Red Hat Update Infrastructure: authentication failure for "/pulp/repos/content/dist/rhel/rhui/server/6/6Server/x86_64/os/repodata/repomd.xml": Password Mismatch I wasn't able to reproduce this. I sync'd my CDS successfully. Then I generated an identity certificate that had expired, and readded the CDS. I then got the password mismatch error. I then generated an identity certificate that had not expired, and resynced the CDS successfully (I didn't even have to delete/readd the CDS). Please try to reproduce the error again after verifying that you've updated the identity certificate. If it still occurs, can you attach /var/log/gofer/agent.log from the CDS, /var/log/httpd/ssl_error_log from the RHUA, and /etc/pki/rhui/identity.crt from the RHUA. Thanks. After expiration of entitlement ca cert and identity cert, I planned to add valid entitlement ca certs and identity cert for both and then the cds sync fails. Here even the entitlement ca cert was also expired along with identity cert. Both had a validity for 1 day and upon recreation of both the valid certs cds sync fails. I've run through this with both an expired entitlement ca and expired identity certificate and can not reproduce any issue. When either (or both) are expired, the CDS sync failes, when neither are expired, the CDS sync succeeds. One thing to note that might be causing the issue: You can not just replace the identity certificate at /etc/pki/rhui/identity.crt with a hand crafted one. The certificate can only be created via rhui-manager, this is because there is an extra step that configures every repo to use this certificate for authentication. The user is never supposed to generate an identity certificate outside of rhui-manager. If you still have the issue please attach the requested files. /var/log/gofer/agent.log from the CDS, /var/log/httpd/ssl_error_log from the RHUA, /etc/pki/rhui/identity.crt from the RHUA, /etc/pki/rhui/entitlement-ca.crt from the RHUA. I could reproduce this issue. For all my setups I use RHUA : ca2.crt CDS : ca1.crt Entitlement CA : ca3.crt When certs expired, I had created a) entitlementca valid for days 1 b) identity cert valid for days 1 the below status after 1 day [root@ip-10-224-6-47 ~]# rhui-manager status CDS1_97_178 ................................................. [ UP ] CDS1_97_178 ................................................. [ ERROR ] CDS2_57_70 .................................................. [ UP ] CDS2_57_70 .................................................. [ ERROR ] Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-x86_64) ... [ SUCCESS ] Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-x86_64) ... [ SUCCESS ] Identity certificate expiration date = 2012-02-21T13:53:36Z .......... [ ERROR ] Identity certificate at /etc/pki/rhui/identity.crt is expired! Entitlement CA certificate expiration date = 2012-02-21T13:46:06Z .... [ ERROR ] Entitlement CA certificate at /etc/pki/rhui/entitlement-ca.crt is expired! 82 After that I moved to /etc/pki/rhui and moved all the certs to a dir old Which made the rhui-manager prompt me for new entitlement ca cert and identity cert Created a valid new entitlement ca cert and updated it. The below status regarding the certs now is [root@ip-10-224-6-47 ~]# rhui-manager status CDS1_97_178 ................................................. [ UP ] CDS1_97_178 ................................................. [ ERROR ] CDS2_57_70 .................................................. [ UP ] CDS2_57_70 .................................................. [ ERROR ] Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-x86_64) ... [ SUCCESS ] Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-x86_64) ... [ SUCCESS ] Identity certificate expiration date = 2013-02-21T06:58:28Z .......... [ OK ] Entitlement CA certificate expiration date = 2013-02-21T06:57:41Z .... [ OK ] 2 ------------------------------------------------------------------------------
-= Red Hat Update Infrastructure Management Tool =-
-= CDS Synchronization Status =-
Last Refreshed: 02:28:12
(updated every 5 seconds, ctrl+c to exit)
CDS1_97_178 ................................................. [ UP ]
CDS2_57_70 .................................................. [ UP ]
Next Sync Last Sync Last Result
------------------------------------------------------------------------------
CDS1_97_178
02-22-2012 03:09 02-22-2012 01:59 Error
CDS2_57_70
02-22-2012 03:10 02-22-2012 01:59 Error
Connected: ip-10-224-6-47.ec2.internal
Created attachment 564853 [details]
logs from gofer ssl and certs output
Tried to sync at 01:59 , but failed , also captured logs for the same time.
The problem is the deletion of the identity certificate from /etc/pki/rhui. From what I can tell, the identity cert generation that happens the very first time you start rhui-manager is not meant to be a means to regenerate the cert; only generate it the first time. That initial generation does not reconfigure any repos to use the new identity cert because there should be no repos since it's the very first time you've run rhui-manager. To restore the RHUI to a functioning state, you could now go into rhui-manager and use the "i" screen to regenerate the identity cert, and it should fix the issue. That being said, I think we can protect against this situation. The fix is easy, we just need to reconfigure any repos if they happen to exist. committed to cloude master: a3bf1275b0faeaa93723b057629c530c2d9b9226 Built into new rh-rhui-tools package at: http://download.devel.redhat.com/brewroot/packages/rh-rhui-tools/2.0.57/1.el6/noarch/rh-rhui-tools-2.0.57-1.el6.noarch.rpm Will request a new iso build later today. appears fixed, will be moved to verified with new ISO
------------------------------------------------------------------------------
-= Red Hat Update Infrastructure Management Tool =-
-= CDS Synchronization Status =-
Last Refreshed: 07:50:00
(updated every 5 seconds, ctrl+c to exit)
CDS1_97_178 ................................................. [ UP ]
CDS2_57_70 .................................................. [ UP ]
Next Sync Last Sync Last Result
------------------------------------------------------------------------------
CDS1_97_178
02-22-2012 09:09 02-22-2012 07:39 Success
CDS2_57_70
02-22-2012 09:10 02-22-2012 07:39 Success
Connected: ip-10-224-6-47.ec2.internal
package included in new iso: http://download.lab.bos.redhat.com/devel/candidates/RHEL-6.2-RHUI-2.0.2-20120222.0/2.0.2/Server/x86_64/iso/RHEL-6.2-RHUI-2.0.2-20120222.0-Server-x86_64-DVD1.iso ------------------------------------------------------------------------------
-= Red Hat Update Infrastructure Management Tool =-
-= CDS Synchronization Status =-
Last Refreshed: 04:21:42
(updated every 5 seconds, ctrl+c to exit)
CDS1_50_75 .................................................. [ UP ]
CDS2_27_41 .................................................. [ UP ]
Next Sync Last Sync Last Result
------------------------------------------------------------------------------
CDS1_50_75
02-24-2012 10:09 02-24-2012 04:09 Error
CDS2_27_41
02-24-2012 07:44 02-24-2012 01:44 Success
Connected: ip-10-98-9-150.ec2.internal
[root@ip-10-98-9-150 rhui]# rhui-manager status
Previous authentication credentials could not be found. Logging into
the RHUI.
If this is the first time using the RHUI, it is recommended to change
the user's password in the User Management section of RHUI Tools.
RHUI Username: admin
RHUI Password:
CDS1_50_75 .................................................. [ UP ]
CDS1_50_75 .................................................. [ SUCCESS ]
CDS2_27_41 .................................................. [ UP ]
CDS2_27_41 .................................................. [ SUCCESS ]
Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-x86_64) ... [ SUCCESS ]
Red Hat Update Infrastructure 2.0 (RPMs) (6Server-x86_64) ............. [ SUCCESS ]
Identity certificate expiration date = 2013-02-23T09:20:09Z .......... [ OK ]
Entitlement CA certificate expiration date = 2012-07-25T16:36:58Z .... [ OK ]
0
Released in RHUI 2.0.2 |