Bug 1262996 - ipa vault internal error on replica without KRA
ipa vault internal error on replica without KRA
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa (Show other bugs)
7.2
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: IPA Maintainers
Namita Soman
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-14 16:41 EDT by Scott Poore
Modified: 2015-11-19 07:06 EST (History)
8 users (show)

See Also:
Fixed In Version: ipa-4.2.0-14.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-19 07:06:38 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Scott Poore 2015-09-14 16:41:27 EDT
Description of problem:

Trying to run vault-retrieve on an IPA Replica without KRA installed is resulting in an internal error.  If I install KRA, it then works.

# ON MASTER with KRA installed:

[root@master ~]# ipa vault-add vupgrade1 --type symmetric --password=Pa55w0rd1
-----------------------
Added vault "vupgrade1"
-----------------------
  Vault name: vupgrade1
  Type: symmetric
  Salt: z+NweI/Kodi1t4SgNY9v3Q==
  Owner users: admin
  Vault user: admin
[root@master ~]# SECRET="$(echo Secret123|base64)"

[root@master ~]# ipa vault-archive vupgrade1 --password='Pa55w0rd1' --data="$SECRET"
------------------------------------
Archived data into vault "vupgrade1"
------------------------------------

# ON REPLICA:

[root@replica ~]# kinit admin
Password for admin@TESTRELM.TEST: 
[root@replica ~]# ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out
ipa: ERROR: an internal error has occurred

Version-Release number of selected component (if applicable):

ipa-server-4.2.0-9.el7.x86_64

How reproducible:
always.

Steps to Reproduce:
1.  Install IPA Master with KRA

ipa-server-install
ipa-kra-install

2.  Install IPA Replica without KRA

ipa-replica-prepare # on master
ipa-replica-install

3.  Create Vault with data on Master

ipa vault-add vupgrade1 --type symmetric --password=Pa55w0rd1
SECRET="$(echo Secret123|base64)"
ipa vault-archive vupgrade1 --password='Pa55w0rd1' --data="$SECRET"


4.  Retrieve data from vault on Replica

ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out

Actual results:

[root@replica ~]# ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out
ipa: ERROR: an internal error has occurred

Expected results:

No error.  Should see output written to file.

Additional info:

Installing KRA on Replica fixes this issue.  But, should not be necessary from what I understand.

httpd/error_log:

[Mon Sep 14 15:25:57.806201 2015] [:error] [pid 5381] ipa: ERROR: non-public: IOError: [Errno 2] No such file or directory
[Mon Sep 14 15:25:57.806215 2015] [:error] [pid 5381] Traceback (most recent call last):
[Mon Sep 14 15:25:57.806217 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 347, in wsgi_execute
[Mon Sep 14 15:25:57.806219 2015] [:error] [pid 5381]     result = self.Command[name](*args, **options)
[Mon Sep 14 15:25:57.806220 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
[Mon Sep 14 15:25:57.806221 2015] [:error] [pid 5381]     ret = self.run(*args, **options)
[Mon Sep 14 15:25:57.806223 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 760, in run
[Mon Sep 14 15:25:57.806224 2015] [:error] [pid 5381]     return self.execute(*args, **options)
[Mon Sep 14 15:25:57.806225 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/ipalib/plugins/vault.py", line 1170, in execute
[Mon Sep 14 15:25:57.806227 2015] [:error] [pid 5381]     transport_cert = kra_client.system_certs.get_transport_cert()
[Mon Sep 14 15:25:57.806228 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/pki/__init__.py", line 298, in handler
[Mon Sep 14 15:25:57.806230 2015] [:error] [pid 5381]     return fn_call(inst, *args, **kwargs)
[Mon Sep 14 15:25:57.806231 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/pki/systemcert.py", line 52, in get_transport_cert
[Mon Sep 14 15:25:57.806232 2015] [:error] [pid 5381]     response = self.connection.get(url, self.headers)
[Mon Sep 14 15:25:57.806234 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/pki/client.py", line 115, in get
[Mon Sep 14 15:25:57.806235 2015] [:error] [pid 5381]     data=payload)
[Mon Sep 14 15:25:57.806236 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 476, in get
[Mon Sep 14 15:25:57.806238 2015] [:error] [pid 5381]     return self.request('GET', url, **kwargs)
[Mon Sep 14 15:25:57.806239 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
[Mon Sep 14 15:25:57.806240 2015] [:error] [pid 5381]     resp = self.send(prep, **send_kwargs)
[Mon Sep 14 15:25:57.806242 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
[Mon Sep 14 15:25:57.806243 2015] [:error] [pid 5381]     r = adapter.send(request, **kwargs)
[Mon Sep 14 15:25:57.806244 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
[Mon Sep 14 15:25:57.806246 2015] [:error] [pid 5381]     timeout=timeout
[Mon Sep 14 15:25:57.806247 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 544, in urlopen
[Mon Sep 14 15:25:57.806248 2015] [:error] [pid 5381]     body=body, headers=headers)
[Mon Sep 14 15:25:57.806249 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 341, in _make_request
[Mon Sep 14 15:25:57.806251 2015] [:error] [pid 5381]     self._validate_conn(conn)
[Mon Sep 14 15:25:57.806253 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 762, in _validate_conn
[Mon Sep 14 15:25:57.806254 2015] [:error] [pid 5381]     conn.connect()
[Mon Sep 14 15:25:57.806255 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 238, in connect
[Mon Sep 14 15:25:57.806257 2015] [:error] [pid 5381]     ssl_version=resolved_ssl_version)
[Mon Sep 14 15:25:57.806258 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/urllib3/util/ssl_.py", line 254, in ssl_wrap_socket
[Mon Sep 14 15:25:57.806259 2015] [:error] [pid 5381]     context.load_cert_chain(certfile, keyfile)
[Mon Sep 14 15:25:57.806261 2015] [:error] [pid 5381] IOError: [Errno 2] No such file or directory
[Mon Sep 14 15:25:57.806472 2015] [:error] [pid 5381] ipa: INFO: [jsonserver_kerb] admin@TESTRELM.TEST: vaultconfig_show(all=False, raw=False, version=u'2.155'): IOError
Comment 2 Endi Sukma Dewata 2015-09-14 19:09:26 EDT
Based on the initial investigation there seems to be several issues:

1. The vault archival/retrieval fails because the KRA agent certificate PEM file (/etc/httpd/alias/kra-agent.pem) is missing. The HTTPD error log will show "No such file or directory" message.

Currently the certificate is only exported to a PEM file during KRA installation (see krainstance.py) and during cert renewal (see renew_ra_cert). So if the certificate is never renewed, IPA replicas that do not have a local KRA will not have the PEM file.

The workaround is to export the certificate manually:
$ pki -d /etc/httpd/alias -C /etc/httpd/alias/pwdfile.txt client-cert-show ipaCert --client-cert /etc/httpd/alias/kra-agent.pem
$ chown root.apache /etc/httpd/alias/kra-agent.pem
$ chmod 660 /etc/httpd/alias/kra-agent.pem

A possible solution is to export the PEM file on-demand and keep it as a cache (see https://fedorahosted.org/freeipa/ticket/5253#comment:1).

2. With the workaround for issue #1, the vault archival/retrieval still fails because the api.env.kra_host seems to return the localhost instead of the actual KRA hostname. The HTTPD error log will show "HTTPError: 404 Client Error: Not Found" message.

The workaround is to explicitly specify the kra_host in /etc/pki/default.conf:
kra_host=master.testrelm.test

3. With the workarounds for issues #1 and #2, the vault archival/retrieval may still fail since api.env.kra_host is a constant, so existing replicas may not be aware of the new KRA location immediately.

The workaround is to restart all replicas after KRA installation.
Comment 3 Scott Poore 2015-09-14 19:21:15 EDT
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/5302
Comment 5 Petr Vobornik 2015-09-15 04:14:45 EDT
Is this really expected to work?

Setting it in env won't work. Better is to get the kra host from ldap - like in kra_is_enabled. 

Requiring KRA backend installed on a master might be other option - probably a short term solution. Effectively that would mean that KRA would have to be installed on each replica because we don't know which replica the client will contact.
Comment 6 Scott Poore 2015-09-15 10:27:55 EDT
Switching Keywords from Regression to TestBlocker since this is a new feature in RHEL7.2.
Comment 9 Scott Poore 2015-10-08 20:28:55 EDT
Verified.  

Version ::

ipa-server-4.2.0-13.el7.x86_64

Results ::

############MASTER

[root@rhel7-1 ~]# ipa vault-add vupgrade1 --type symmetric --password=Pa55w0rd1
-----------------------
Added vault "vupgrade1"
-----------------------
  Vault name: vupgrade1
  Type: symmetric
  Salt: VHLTRHQUmvMTxLi6JWq4+w==
  Owner users: admin
  Vault user: admin

[root@rhel7-1 ~]# SECRET="$(echo Secret123|base64)"

[root@rhel7-1 ~]# ipa vault-archive vupgrade1 --password='Pa55w0rd1' --data="$SECRET"
------------------------------------
Archived data into vault "vupgrade1"
------------------------------------

[root@rhel7-1 ~]# 


#############REPLICA with CA without KRA

[root@rhel7-2 ~]# kinit admin
Password for admin@EXAMPLE.COM: 

[root@rhel7-2 ~]# ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out
-------------------------------------
Retrieved data from vault "vupgrade1"
-------------------------------------

[root@rhel7-2 ~]# cat /tmp/vault.out
Secret123
Comment 10 Jan Cholasta 2015-10-12 08:56:54 EDT
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/5360
Comment 13 Scott Poore 2015-10-12 16:43:40 EDT
Verified.

Version ::

ipa-server-4.2.0-14.el7.x86_64

Results ::

[root@rhel7-1 ~]# ipa vault-add vupgrade --type symmetric --password=Pa55w0rd1
----------------------
Added vault "vupgrade"
----------------------
  Vault name: vupgrade
  Type: symmetric
  Salt: rk3dXx7wROCyPWQY5oB8Cw==
  Owner users: admin
  Vault user: admin

[root@rhel7-1 ~]# ipa vault-archive vupgrade --password='Pa55w0rd1' --data="$(echo Secret123|base64)"
-----------------------------------
Archived data into vault "vupgrade"
-----------------------------------

[root@rhel7-2 ~]# ipa vault-retrieve vupgrade --password='Pa55w0rd1' --out=/tmp/vault.out
------------------------------------
Retrieved data from vault "vupgrade"
------------------------------------

[root@rhel7-2 ~]# cat /tmp/vault.out
Secret123
Comment 14 Scott Poore 2015-10-16 09:31:14 EDT
moving this one back to ON_QA as the fix from comment #11 requires testing an upgrade as well.
Comment 15 Scott Poore 2015-10-16 11:15:54 EDT
Additional verification with upgrade to cover fix from comment #11.

Started with IPA server with ipa-server-4.1.0-18.el7_1.4.x86_64

Upgraded to ipa-server-4.2.0-15.el7.x86_64

Replaced:
  ipa-server.x86_64 0:4.1.0-18.el7_1.4          libipa_hbac-python.x86_64 0:1.12.2-58.el7_1.17         

Complete!
[root@rhel7-1 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

[root@rhel7-1 ~]# grep "KRA is not enabled" /var/log/ipaupgrade.log 
2015-10-16T14:55:24Z INFO KRA is not enabled
Comment 16 Xiyang Dong 2015-11-15 16:32:27 EST
Internal Error shows up when upgrade MASTER from ipa-server.x86_64 0:4.1.0-18.el7 to ipa-server.x86_64 0:4.2.0-15.el7:
 

[root@mgmt7 ~]# ipa-kra-install -p Secret123 -U

===================================================================
This program will setup Dogtag KRA for the IPA Server.


Configuring KRA server (pki-tomcatd). Estimated time: 2 minutes 6 seconds
  [1/8]: configuring KRA instance
  [2/8]: create KRA agent
  [3/8]: restarting KRA
  [4/8]: configure certmonger for renewals
  [5/8]: configure certificate renewals
  [6/8]: configure HTTP to proxy connections
  [7/8]: add vault container
  [8/8]: apply LDAP updates
Done configuring KRA server (pki-tomcatd).
Restarting the directory server
The ipa-kra-install command was successful

[root@mgmt7 ~]# kinit admin
Password for admin@TESTRELM.TEST: 

[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: an internal error has occurred

[root@mgmt7 ~]# echo Secret123|base64
U2VjcmV0MTIzCg==

[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred

[root@mgmt7 ~]# grep "KRA is not enabled" /var/log/ipaupgrade.log
2015-11-15T20:18:45Z INFO KRA is not enabled

[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: vault with name "vupgrade" already exists

[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred


From /var/log/httpd/error_log:
.
.
.
[Sun Nov 15 16:27:16.261426 2015] [:error] [pid 20785] ipa: ERROR: non-public: SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261468 2015] [:error] [pid 20785] Traceback (most recent call last):
[Sun Nov 15 16:27:16.261475 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 347, in wsgi_execute
[Sun Nov 15 16:27:16.261502 2015] [:error] [pid 20785]     result = self.Command[name](*args, **options)
[Sun Nov 15 16:27:16.261509 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
[Sun Nov 15 16:27:16.261515 2015] [:error] [pid 20785]     ret = self.run(*args, **options)
[Sun Nov 15 16:27:16.261521 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 760, in run
[Sun Nov 15 16:27:16.261527 2015] [:error] [pid 20785]     return self.execute(*args, **options)
[Sun Nov 15 16:27:16.261533 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/plugins/vault.py", line 1471, in execute
[Sun Nov 15 16:27:16.261539 2015] [:error] [pid 20785]     transport_cert = kra_client.system_certs.get_transport_cert()
[Sun Nov 15 16:27:16.261545 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/__init__.py", line 298, in handler
[Sun Nov 15 16:27:16.261552 2015] [:error] [pid 20785]     return fn_call(inst, *args, **kwargs)
[Sun Nov 15 16:27:16.261558 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/systemcert.py", line 52, in get_transport_cert
[Sun Nov 15 16:27:16.261564 2015] [:error] [pid 20785]     response = self.connection.get(url, self.headers)
[Sun Nov 15 16:27:16.261570 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/client.py", line 115, in get
[Sun Nov 15 16:27:16.261576 2015] [:error] [pid 20785]     data=payload)
[Sun Nov 15 16:27:16.261582 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 319, in get
[Sun Nov 15 16:27:16.261588 2015] [:error] [pid 20785]     return self.request('GET', url, **kwargs)
[Sun Nov 15 16:27:16.261593 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 288, in request
[Sun Nov 15 16:27:16.261600 2015] [:error] [pid 20785]     resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
[Sun Nov 15 16:27:16.261606 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 383, in send
[Sun Nov 15 16:27:16.261612 2015] [:error] [pid 20785]     r = adapter.send(request, **kwargs)
[Sun Nov 15 16:27:16.261617 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 213, in send
[Sun Nov 15 16:27:16.261623 2015] [:error] [pid 20785]     raise SSLError(e)
[Sun Nov 15 16:27:16.261629 2015] [:error] [pid 20785] SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261937 2015] [:error] [pid 20785] ipa: INFO: [jsonserver_session] admin@TESTRELM.TEST: vaultconfig_show(all=False, raw=False, version=u'2.156'): SSLError
Comment 17 Xiyang Dong 2015-11-17 15:28:59 EST
opened https://bugzilla.redhat.com/show_bug.cgi?id=1282935, put bz back to verified
Comment 18 errata-xmlrpc 2015-11-19 07:06:38 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2362.html

Note You need to log in before you can comment on or make changes to this bug.