Bug 1262996
| Summary: | ipa vault internal error on replica without KRA | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Scott Poore <spoore> |
| Component: | ipa | Assignee: | IPA Maintainers <ipa-maint> |
| Status: | CLOSED ERRATA | QA Contact: | Namita Soman <nsoman> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.2 | CC: | edewata, jcholast, ksiddiqu, mkosek, pvoborni, rcritten, tlavigne, xdong |
| Target Milestone: | rc | Keywords: | TestBlocker |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ipa-4.2.0-14.el7 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-11-19 12:06:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Based on the initial investigation there seems to be several issues: 1. The vault archival/retrieval fails because the KRA agent certificate PEM file (/etc/httpd/alias/kra-agent.pem) is missing. The HTTPD error log will show "No such file or directory" message. Currently the certificate is only exported to a PEM file during KRA installation (see krainstance.py) and during cert renewal (see renew_ra_cert). So if the certificate is never renewed, IPA replicas that do not have a local KRA will not have the PEM file. The workaround is to export the certificate manually: $ pki -d /etc/httpd/alias -C /etc/httpd/alias/pwdfile.txt client-cert-show ipaCert --client-cert /etc/httpd/alias/kra-agent.pem $ chown root.apache /etc/httpd/alias/kra-agent.pem $ chmod 660 /etc/httpd/alias/kra-agent.pem A possible solution is to export the PEM file on-demand and keep it as a cache (see https://fedorahosted.org/freeipa/ticket/5253#comment:1). 2. With the workaround for issue #1, the vault archival/retrieval still fails because the api.env.kra_host seems to return the localhost instead of the actual KRA hostname. The HTTPD error log will show "HTTPError: 404 Client Error: Not Found" message. The workaround is to explicitly specify the kra_host in /etc/pki/default.conf: kra_host=master.testrelm.test 3. With the workarounds for issues #1 and #2, the vault archival/retrieval may still fail since api.env.kra_host is a constant, so existing replicas may not be aware of the new KRA location immediately. The workaround is to restart all replicas after KRA installation. Upstream ticket: https://fedorahosted.org/freeipa/ticket/5302 Is this really expected to work? Setting it in env won't work. Better is to get the kra host from ldap - like in kra_is_enabled. Requiring KRA backend installed on a master might be other option - probably a short term solution. Effectively that would mean that KRA would have to be installed on each replica because we don't know which replica the client will contact. Switching Keywords from Regression to TestBlocker since this is a new feature in RHEL7.2. Fixed upstream master: https://fedorahosted.org/freeipa/changeset/b035a2a11442c190dc68d9e653b98ef396332c8e https://fedorahosted.org/freeipa/changeset/4b381b1503d8c282b4d4680beed2a9439f5b61cc ipa-4-2: https://fedorahosted.org/freeipa/changeset/10020525eb54e36e55b2788114d68106fc995db6 https://fedorahosted.org/freeipa/changeset/0cfa43456e224fa919bc74155242a34e64152432 Verified. Version :: ipa-server-4.2.0-13.el7.x86_64 Results :: ############MASTER [root@rhel7-1 ~]# ipa vault-add vupgrade1 --type symmetric --password=Pa55w0rd1 ----------------------- Added vault "vupgrade1" ----------------------- Vault name: vupgrade1 Type: symmetric Salt: VHLTRHQUmvMTxLi6JWq4+w== Owner users: admin Vault user: admin [root@rhel7-1 ~]# SECRET="$(echo Secret123|base64)" [root@rhel7-1 ~]# ipa vault-archive vupgrade1 --password='Pa55w0rd1' --data="$SECRET" ------------------------------------ Archived data into vault "vupgrade1" ------------------------------------ [root@rhel7-1 ~]# #############REPLICA with CA without KRA [root@rhel7-2 ~]# kinit admin Password for admin: [root@rhel7-2 ~]# ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out ------------------------------------- Retrieved data from vault "vupgrade1" ------------------------------------- [root@rhel7-2 ~]# cat /tmp/vault.out Secret123 Upstream ticket: https://fedorahosted.org/freeipa/ticket/5360 Fixed upstream master: https://fedorahosted.org/freeipa/changeset/61bdbd6e47b2cd2a62f7e50a6a6cbd2e272470d9 ipa-4-2: https://fedorahosted.org/freeipa/changeset/9182f40ac549fc0104878a5599c9effe4f80c3ec Verified. Version :: ipa-server-4.2.0-14.el7.x86_64 Results :: [root@rhel7-1 ~]# ipa vault-add vupgrade --type symmetric --password=Pa55w0rd1 ---------------------- Added vault "vupgrade" ---------------------- Vault name: vupgrade Type: symmetric Salt: rk3dXx7wROCyPWQY5oB8Cw== Owner users: admin Vault user: admin [root@rhel7-1 ~]# ipa vault-archive vupgrade --password='Pa55w0rd1' --data="$(echo Secret123|base64)" ----------------------------------- Archived data into vault "vupgrade" ----------------------------------- [root@rhel7-2 ~]# ipa vault-retrieve vupgrade --password='Pa55w0rd1' --out=/tmp/vault.out ------------------------------------ Retrieved data from vault "vupgrade" ------------------------------------ [root@rhel7-2 ~]# cat /tmp/vault.out Secret123 moving this one back to ON_QA as the fix from comment #11 requires testing an upgrade as well. Additional verification with upgrade to cover fix from comment #11. Started with IPA server with ipa-server-4.1.0-18.el7_1.4.x86_64 Upgraded to ipa-server-4.2.0-15.el7.x86_64 Replaced: ipa-server.x86_64 0:4.1.0-18.el7_1.4 libipa_hbac-python.x86_64 0:1.12.2-58.el7_1.17 Complete! [root@rhel7-1 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING ipa_memcached Service: RUNNING httpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful [root@rhel7-1 ~]# grep "KRA is not enabled" /var/log/ipaupgrade.log 2015-10-16T14:55:24Z INFO KRA is not enabled Internal Error shows up when upgrade MASTER from ipa-server.x86_64 0:4.1.0-18.el7 to ipa-server.x86_64 0:4.2.0-15.el7:
[root@mgmt7 ~]# ipa-kra-install -p Secret123 -U
===================================================================
This program will setup Dogtag KRA for the IPA Server.
Configuring KRA server (pki-tomcatd). Estimated time: 2 minutes 6 seconds
[1/8]: configuring KRA instance
[2/8]: create KRA agent
[3/8]: restarting KRA
[4/8]: configure certmonger for renewals
[5/8]: configure certificate renewals
[6/8]: configure HTTP to proxy connections
[7/8]: add vault container
[8/8]: apply LDAP updates
Done configuring KRA server (pki-tomcatd).
Restarting the directory server
The ipa-kra-install command was successful
[root@mgmt7 ~]# kinit admin
Password for admin:
[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: an internal error has occurred
[root@mgmt7 ~]# echo Secret123|base64
U2VjcmV0MTIzCg==
[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred
[root@mgmt7 ~]# grep "KRA is not enabled" /var/log/ipaupgrade.log
2015-11-15T20:18:45Z INFO KRA is not enabled
[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: vault with name "vupgrade" already exists
[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred
From /var/log/httpd/error_log:
.
.
.
[Sun Nov 15 16:27:16.261426 2015] [:error] [pid 20785] ipa: ERROR: non-public: SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261468 2015] [:error] [pid 20785] Traceback (most recent call last):
[Sun Nov 15 16:27:16.261475 2015] [:error] [pid 20785] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 347, in wsgi_execute
[Sun Nov 15 16:27:16.261502 2015] [:error] [pid 20785] result = self.Command[name](*args, **options)
[Sun Nov 15 16:27:16.261509 2015] [:error] [pid 20785] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
[Sun Nov 15 16:27:16.261515 2015] [:error] [pid 20785] ret = self.run(*args, **options)
[Sun Nov 15 16:27:16.261521 2015] [:error] [pid 20785] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 760, in run
[Sun Nov 15 16:27:16.261527 2015] [:error] [pid 20785] return self.execute(*args, **options)
[Sun Nov 15 16:27:16.261533 2015] [:error] [pid 20785] File "/usr/lib/python2.7/site-packages/ipalib/plugins/vault.py", line 1471, in execute
[Sun Nov 15 16:27:16.261539 2015] [:error] [pid 20785] transport_cert = kra_client.system_certs.get_transport_cert()
[Sun Nov 15 16:27:16.261545 2015] [:error] [pid 20785] File "/usr/lib/python2.7/site-packages/pki/__init__.py", line 298, in handler
[Sun Nov 15 16:27:16.261552 2015] [:error] [pid 20785] return fn_call(inst, *args, **kwargs)
[Sun Nov 15 16:27:16.261558 2015] [:error] [pid 20785] File "/usr/lib/python2.7/site-packages/pki/systemcert.py", line 52, in get_transport_cert
[Sun Nov 15 16:27:16.261564 2015] [:error] [pid 20785] response = self.connection.get(url, self.headers)
[Sun Nov 15 16:27:16.261570 2015] [:error] [pid 20785] File "/usr/lib/python2.7/site-packages/pki/client.py", line 115, in get
[Sun Nov 15 16:27:16.261576 2015] [:error] [pid 20785] data=payload)
[Sun Nov 15 16:27:16.261582 2015] [:error] [pid 20785] File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 319, in get
[Sun Nov 15 16:27:16.261588 2015] [:error] [pid 20785] return self.request('GET', url, **kwargs)
[Sun Nov 15 16:27:16.261593 2015] [:error] [pid 20785] File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 288, in request
[Sun Nov 15 16:27:16.261600 2015] [:error] [pid 20785] resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
[Sun Nov 15 16:27:16.261606 2015] [:error] [pid 20785] File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 383, in send
[Sun Nov 15 16:27:16.261612 2015] [:error] [pid 20785] r = adapter.send(request, **kwargs)
[Sun Nov 15 16:27:16.261617 2015] [:error] [pid 20785] File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 213, in send
[Sun Nov 15 16:27:16.261623 2015] [:error] [pid 20785] raise SSLError(e)
[Sun Nov 15 16:27:16.261629 2015] [:error] [pid 20785] SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261937 2015] [:error] [pid 20785] ipa: INFO: [jsonserver_session] admin: vaultconfig_show(all=False, raw=False, version=u'2.156'): SSLError
opened https://bugzilla.redhat.com/show_bug.cgi?id=1282935, put bz back to verified Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-2362.html |
Description of problem: Trying to run vault-retrieve on an IPA Replica without KRA installed is resulting in an internal error. If I install KRA, it then works. # ON MASTER with KRA installed: [root@master ~]# ipa vault-add vupgrade1 --type symmetric --password=Pa55w0rd1 ----------------------- Added vault "vupgrade1" ----------------------- Vault name: vupgrade1 Type: symmetric Salt: z+NweI/Kodi1t4SgNY9v3Q== Owner users: admin Vault user: admin [root@master ~]# SECRET="$(echo Secret123|base64)" [root@master ~]# ipa vault-archive vupgrade1 --password='Pa55w0rd1' --data="$SECRET" ------------------------------------ Archived data into vault "vupgrade1" ------------------------------------ # ON REPLICA: [root@replica ~]# kinit admin Password for admin: [root@replica ~]# ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out ipa: ERROR: an internal error has occurred Version-Release number of selected component (if applicable): ipa-server-4.2.0-9.el7.x86_64 How reproducible: always. Steps to Reproduce: 1. Install IPA Master with KRA ipa-server-install ipa-kra-install 2. Install IPA Replica without KRA ipa-replica-prepare # on master ipa-replica-install 3. Create Vault with data on Master ipa vault-add vupgrade1 --type symmetric --password=Pa55w0rd1 SECRET="$(echo Secret123|base64)" ipa vault-archive vupgrade1 --password='Pa55w0rd1' --data="$SECRET" 4. Retrieve data from vault on Replica ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out Actual results: [root@replica ~]# ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out ipa: ERROR: an internal error has occurred Expected results: No error. Should see output written to file. Additional info: Installing KRA on Replica fixes this issue. But, should not be necessary from what I understand. httpd/error_log: [Mon Sep 14 15:25:57.806201 2015] [:error] [pid 5381] ipa: ERROR: non-public: IOError: [Errno 2] No such file or directory [Mon Sep 14 15:25:57.806215 2015] [:error] [pid 5381] Traceback (most recent call last): [Mon Sep 14 15:25:57.806217 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 347, in wsgi_execute [Mon Sep 14 15:25:57.806219 2015] [:error] [pid 5381] result = self.Command[name](*args, **options) [Mon Sep 14 15:25:57.806220 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__ [Mon Sep 14 15:25:57.806221 2015] [:error] [pid 5381] ret = self.run(*args, **options) [Mon Sep 14 15:25:57.806223 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 760, in run [Mon Sep 14 15:25:57.806224 2015] [:error] [pid 5381] return self.execute(*args, **options) [Mon Sep 14 15:25:57.806225 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/ipalib/plugins/vault.py", line 1170, in execute [Mon Sep 14 15:25:57.806227 2015] [:error] [pid 5381] transport_cert = kra_client.system_certs.get_transport_cert() [Mon Sep 14 15:25:57.806228 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/pki/__init__.py", line 298, in handler [Mon Sep 14 15:25:57.806230 2015] [:error] [pid 5381] return fn_call(inst, *args, **kwargs) [Mon Sep 14 15:25:57.806231 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/pki/systemcert.py", line 52, in get_transport_cert [Mon Sep 14 15:25:57.806232 2015] [:error] [pid 5381] response = self.connection.get(url, self.headers) [Mon Sep 14 15:25:57.806234 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/pki/client.py", line 115, in get [Mon Sep 14 15:25:57.806235 2015] [:error] [pid 5381] data=payload) [Mon Sep 14 15:25:57.806236 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 476, in get [Mon Sep 14 15:25:57.806238 2015] [:error] [pid 5381] return self.request('GET', url, **kwargs) [Mon Sep 14 15:25:57.806239 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request [Mon Sep 14 15:25:57.806240 2015] [:error] [pid 5381] resp = self.send(prep, **send_kwargs) [Mon Sep 14 15:25:57.806242 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send [Mon Sep 14 15:25:57.806243 2015] [:error] [pid 5381] r = adapter.send(request, **kwargs) [Mon Sep 14 15:25:57.806244 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send [Mon Sep 14 15:25:57.806246 2015] [:error] [pid 5381] timeout=timeout [Mon Sep 14 15:25:57.806247 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 544, in urlopen [Mon Sep 14 15:25:57.806248 2015] [:error] [pid 5381] body=body, headers=headers) [Mon Sep 14 15:25:57.806249 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 341, in _make_request [Mon Sep 14 15:25:57.806251 2015] [:error] [pid 5381] self._validate_conn(conn) [Mon Sep 14 15:25:57.806253 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 762, in _validate_conn [Mon Sep 14 15:25:57.806254 2015] [:error] [pid 5381] conn.connect() [Mon Sep 14 15:25:57.806255 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 238, in connect [Mon Sep 14 15:25:57.806257 2015] [:error] [pid 5381] ssl_version=resolved_ssl_version) [Mon Sep 14 15:25:57.806258 2015] [:error] [pid 5381] File "/usr/lib/python2.7/site-packages/urllib3/util/ssl_.py", line 254, in ssl_wrap_socket [Mon Sep 14 15:25:57.806259 2015] [:error] [pid 5381] context.load_cert_chain(certfile, keyfile) [Mon Sep 14 15:25:57.806261 2015] [:error] [pid 5381] IOError: [Errno 2] No such file or directory [Mon Sep 14 15:25:57.806472 2015] [:error] [pid 5381] ipa: INFO: [jsonserver_kerb] admin: vaultconfig_show(all=False, raw=False, version=u'2.155'): IOError