RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1262996 - ipa vault internal error on replica without KRA
Summary: ipa vault internal error on replica without KRA
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: Namita Soman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-09-14 20:41 UTC by Scott Poore
Modified: 2015-11-19 12:06 UTC (History)
8 users (show)

Fixed In Version: ipa-4.2.0-14.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-19 12:06:38 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2362 0 normal SHIPPED_LIVE ipa bug fix and enhancement update 2015-11-19 10:40:46 UTC

Description Scott Poore 2015-09-14 20:41:27 UTC 
Description of problem:

Trying to run vault-retrieve on an IPA Replica without KRA installed is resulting in an internal error.  If I install KRA, it then works.

# ON MASTER with KRA installed:

[root@master ~]# ipa vault-add vupgrade1 --type symmetric --password=Pa55w0rd1
-----------------------
Added vault "vupgrade1"
-----------------------
  Vault name: vupgrade1
  Type: symmetric
  Salt: z+NweI/Kodi1t4SgNY9v3Q==
  Owner users: admin
  Vault user: admin
[root@master ~]# SECRET="$(echo Secret123|base64)"

[root@master ~]# ipa vault-archive vupgrade1 --password='Pa55w0rd1' --data="$SECRET"
------------------------------------
Archived data into vault "vupgrade1"
------------------------------------

# ON REPLICA:

[root@replica ~]# kinit admin
Password for admin: 
[root@replica ~]# ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out
ipa: ERROR: an internal error has occurred

Version-Release number of selected component (if applicable):

ipa-server-4.2.0-9.el7.x86_64

How reproducible:
always.

Steps to Reproduce:
1.  Install IPA Master with KRA

ipa-server-install
ipa-kra-install

2.  Install IPA Replica without KRA

ipa-replica-prepare # on master
ipa-replica-install

3.  Create Vault with data on Master

ipa vault-add vupgrade1 --type symmetric --password=Pa55w0rd1
SECRET="$(echo Secret123|base64)"
ipa vault-archive vupgrade1 --password='Pa55w0rd1' --data="$SECRET"


4.  Retrieve data from vault on Replica

ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out

Actual results:

[root@replica ~]# ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out
ipa: ERROR: an internal error has occurred

Expected results:

No error.  Should see output written to file.

Additional info:

Installing KRA on Replica fixes this issue.  But, should not be necessary from what I understand.

httpd/error_log:

[Mon Sep 14 15:25:57.806201 2015] [:error] [pid 5381] ipa: ERROR: non-public: IOError: [Errno 2] No such file or directory
[Mon Sep 14 15:25:57.806215 2015] [:error] [pid 5381] Traceback (most recent call last):
[Mon Sep 14 15:25:57.806217 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 347, in wsgi_execute
[Mon Sep 14 15:25:57.806219 2015] [:error] [pid 5381]     result = self.Command[name](*args, **options)
[Mon Sep 14 15:25:57.806220 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
[Mon Sep 14 15:25:57.806221 2015] [:error] [pid 5381]     ret = self.run(*args, **options)
[Mon Sep 14 15:25:57.806223 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 760, in run
[Mon Sep 14 15:25:57.806224 2015] [:error] [pid 5381]     return self.execute(*args, **options)
[Mon Sep 14 15:25:57.806225 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/ipalib/plugins/vault.py", line 1170, in execute
[Mon Sep 14 15:25:57.806227 2015] [:error] [pid 5381]     transport_cert = kra_client.system_certs.get_transport_cert()
[Mon Sep 14 15:25:57.806228 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/pki/__init__.py", line 298, in handler
[Mon Sep 14 15:25:57.806230 2015] [:error] [pid 5381]     return fn_call(inst, *args, **kwargs)
[Mon Sep 14 15:25:57.806231 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/pki/systemcert.py", line 52, in get_transport_cert
[Mon Sep 14 15:25:57.806232 2015] [:error] [pid 5381]     response = self.connection.get(url, self.headers)
[Mon Sep 14 15:25:57.806234 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/pki/client.py", line 115, in get
[Mon Sep 14 15:25:57.806235 2015] [:error] [pid 5381]     data=payload)
[Mon Sep 14 15:25:57.806236 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 476, in get
[Mon Sep 14 15:25:57.806238 2015] [:error] [pid 5381]     return self.request('GET', url, **kwargs)
[Mon Sep 14 15:25:57.806239 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
[Mon Sep 14 15:25:57.806240 2015] [:error] [pid 5381]     resp = self.send(prep, **send_kwargs)
[Mon Sep 14 15:25:57.806242 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
[Mon Sep 14 15:25:57.806243 2015] [:error] [pid 5381]     r = adapter.send(request, **kwargs)
[Mon Sep 14 15:25:57.806244 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
[Mon Sep 14 15:25:57.806246 2015] [:error] [pid 5381]     timeout=timeout
[Mon Sep 14 15:25:57.806247 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 544, in urlopen
[Mon Sep 14 15:25:57.806248 2015] [:error] [pid 5381]     body=body, headers=headers)
[Mon Sep 14 15:25:57.806249 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 341, in _make_request
[Mon Sep 14 15:25:57.806251 2015] [:error] [pid 5381]     self._validate_conn(conn)
[Mon Sep 14 15:25:57.806253 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 762, in _validate_conn
[Mon Sep 14 15:25:57.806254 2015] [:error] [pid 5381]     conn.connect()
[Mon Sep 14 15:25:57.806255 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 238, in connect
[Mon Sep 14 15:25:57.806257 2015] [:error] [pid 5381]     ssl_version=resolved_ssl_version)
[Mon Sep 14 15:25:57.806258 2015] [:error] [pid 5381]   File "/usr/lib/python2.7/site-packages/urllib3/util/ssl_.py", line 254, in ssl_wrap_socket
[Mon Sep 14 15:25:57.806259 2015] [:error] [pid 5381]     context.load_cert_chain(certfile, keyfile)
[Mon Sep 14 15:25:57.806261 2015] [:error] [pid 5381] IOError: [Errno 2] No such file or directory
[Mon Sep 14 15:25:57.806472 2015] [:error] [pid 5381] ipa: INFO: [jsonserver_kerb] admin: vaultconfig_show(all=False, raw=False, version=u'2.155'): IOError
Comment 2 Endi Sukma Dewata 2015-09-14 23:09:26 UTC 
Based on the initial investigation there seems to be several issues:

1. The vault archival/retrieval fails because the KRA agent certificate PEM file (/etc/httpd/alias/kra-agent.pem) is missing. The HTTPD error log will show "No such file or directory" message.

Currently the certificate is only exported to a PEM file during KRA installation (see krainstance.py) and during cert renewal (see renew_ra_cert). So if the certificate is never renewed, IPA replicas that do not have a local KRA will not have the PEM file.

The workaround is to export the certificate manually:
$ pki -d /etc/httpd/alias -C /etc/httpd/alias/pwdfile.txt client-cert-show ipaCert --client-cert /etc/httpd/alias/kra-agent.pem
$ chown root.apache /etc/httpd/alias/kra-agent.pem
$ chmod 660 /etc/httpd/alias/kra-agent.pem

A possible solution is to export the PEM file on-demand and keep it as a cache (see https://fedorahosted.org/freeipa/ticket/5253#comment:1).

2. With the workaround for issue #1, the vault archival/retrieval still fails because the api.env.kra_host seems to return the localhost instead of the actual KRA hostname. The HTTPD error log will show "HTTPError: 404 Client Error: Not Found" message.

The workaround is to explicitly specify the kra_host in /etc/pki/default.conf:
kra_host=master.testrelm.test

3. With the workarounds for issues #1 and #2, the vault archival/retrieval may still fail since api.env.kra_host is a constant, so existing replicas may not be aware of the new KRA location immediately.

The workaround is to restart all replicas after KRA installation.
Comment 3 Scott Poore 2015-09-14 23:21:15 UTC 
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/5302
Comment 5 Petr Vobornik 2015-09-15 08:14:45 UTC 
Is this really expected to work?

Setting it in env won't work. Better is to get the kra host from ldap - like in kra_is_enabled. 

Requiring KRA backend installed on a master might be other option - probably a short term solution. Effectively that would mean that KRA would have to be installed on each replica because we don't know which replica the client will contact.
Comment 6 Scott Poore 2015-09-15 14:27:55 UTC 
Switching Keywords from Regression to TestBlocker since this is a new feature in RHEL7.2.
Comment 7 Jan Cholasta 2015-10-08 11:54:58 UTC 
Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/b035a2a11442c190dc68d9e653b98ef396332c8e
https://fedorahosted.org/freeipa/changeset/4b381b1503d8c282b4d4680beed2a9439f5b61cc
ipa-4-2:
https://fedorahosted.org/freeipa/changeset/10020525eb54e36e55b2788114d68106fc995db6
https://fedorahosted.org/freeipa/changeset/0cfa43456e224fa919bc74155242a34e64152432
Comment 9 Scott Poore 2015-10-09 00:28:55 UTC 
Verified.  

Version ::

ipa-server-4.2.0-13.el7.x86_64

Results ::

############MASTER

[root@rhel7-1 ~]# ipa vault-add vupgrade1 --type symmetric --password=Pa55w0rd1
-----------------------
Added vault "vupgrade1"
-----------------------
  Vault name: vupgrade1
  Type: symmetric
  Salt: VHLTRHQUmvMTxLi6JWq4+w==
  Owner users: admin
  Vault user: admin

[root@rhel7-1 ~]# SECRET="$(echo Secret123|base64)"

[root@rhel7-1 ~]# ipa vault-archive vupgrade1 --password='Pa55w0rd1' --data="$SECRET"
------------------------------------
Archived data into vault "vupgrade1"
------------------------------------

[root@rhel7-1 ~]# 


#############REPLICA with CA without KRA

[root@rhel7-2 ~]# kinit admin
Password for admin: 

[root@rhel7-2 ~]# ipa vault-retrieve vupgrade1 --password='Pa55w0rd1' --out=/tmp/vault.out
-------------------------------------
Retrieved data from vault "vupgrade1"
-------------------------------------

[root@rhel7-2 ~]# cat /tmp/vault.out
Secret123
Comment 10 Jan Cholasta 2015-10-12 12:56:54 UTC 
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/5360
Comment 11 Jan Cholasta 2015-10-12 13:51:56 UTC 
Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/61bdbd6e47b2cd2a62f7e50a6a6cbd2e272470d9
ipa-4-2:
https://fedorahosted.org/freeipa/changeset/9182f40ac549fc0104878a5599c9effe4f80c3ec
Comment 13 Scott Poore 2015-10-12 20:43:40 UTC 
Verified.

Version ::

ipa-server-4.2.0-14.el7.x86_64

Results ::

[root@rhel7-1 ~]# ipa vault-add vupgrade --type symmetric --password=Pa55w0rd1
----------------------
Added vault "vupgrade"
----------------------
  Vault name: vupgrade
  Type: symmetric
  Salt: rk3dXx7wROCyPWQY5oB8Cw==
  Owner users: admin
  Vault user: admin

[root@rhel7-1 ~]# ipa vault-archive vupgrade --password='Pa55w0rd1' --data="$(echo Secret123|base64)"
-----------------------------------
Archived data into vault "vupgrade"
-----------------------------------

[root@rhel7-2 ~]# ipa vault-retrieve vupgrade --password='Pa55w0rd1' --out=/tmp/vault.out
------------------------------------
Retrieved data from vault "vupgrade"
------------------------------------

[root@rhel7-2 ~]# cat /tmp/vault.out
Secret123
Comment 14 Scott Poore 2015-10-16 13:31:14 UTC 
moving this one back to ON_QA as the fix from comment #11 requires testing an upgrade as well.
Comment 15 Scott Poore 2015-10-16 15:15:54 UTC 
Additional verification with upgrade to cover fix from comment #11.

Started with IPA server with ipa-server-4.1.0-18.el7_1.4.x86_64

Upgraded to ipa-server-4.2.0-15.el7.x86_64

Replaced:
  ipa-server.x86_64 0:4.1.0-18.el7_1.4          libipa_hbac-python.x86_64 0:1.12.2-58.el7_1.17         

Complete!
[root@rhel7-1 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

[root@rhel7-1 ~]# grep "KRA is not enabled" /var/log/ipaupgrade.log 
2015-10-16T14:55:24Z INFO KRA is not enabled
Comment 16 Xiyang Dong 2015-11-15 21:32:27 UTC 
Internal Error shows up when upgrade MASTER from ipa-server.x86_64 0:4.1.0-18.el7 to ipa-server.x86_64 0:4.2.0-15.el7:
 

[root@mgmt7 ~]# ipa-kra-install -p Secret123 -U

===================================================================
This program will setup Dogtag KRA for the IPA Server.


Configuring KRA server (pki-tomcatd). Estimated time: 2 minutes 6 seconds
  [1/8]: configuring KRA instance
  [2/8]: create KRA agent
  [3/8]: restarting KRA
  [4/8]: configure certmonger for renewals
  [5/8]: configure certificate renewals
  [6/8]: configure HTTP to proxy connections
  [7/8]: add vault container
  [8/8]: apply LDAP updates
Done configuring KRA server (pki-tomcatd).
Restarting the directory server
The ipa-kra-install command was successful

[root@mgmt7 ~]# kinit admin
Password for admin: 

[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: an internal error has occurred

[root@mgmt7 ~]# echo Secret123|base64
U2VjcmV0MTIzCg==

[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred

[root@mgmt7 ~]# grep "KRA is not enabled" /var/log/ipaupgrade.log
2015-11-15T20:18:45Z INFO KRA is not enabled

[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: vault with name "vupgrade" already exists

[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred


From /var/log/httpd/error_log:
.
.
.
[Sun Nov 15 16:27:16.261426 2015] [:error] [pid 20785] ipa: ERROR: non-public: SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261468 2015] [:error] [pid 20785] Traceback (most recent call last):
[Sun Nov 15 16:27:16.261475 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 347, in wsgi_execute
[Sun Nov 15 16:27:16.261502 2015] [:error] [pid 20785]     result = self.Command[name](*args, **options)
[Sun Nov 15 16:27:16.261509 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
[Sun Nov 15 16:27:16.261515 2015] [:error] [pid 20785]     ret = self.run(*args, **options)
[Sun Nov 15 16:27:16.261521 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 760, in run
[Sun Nov 15 16:27:16.261527 2015] [:error] [pid 20785]     return self.execute(*args, **options)
[Sun Nov 15 16:27:16.261533 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/plugins/vault.py", line 1471, in execute
[Sun Nov 15 16:27:16.261539 2015] [:error] [pid 20785]     transport_cert = kra_client.system_certs.get_transport_cert()
[Sun Nov 15 16:27:16.261545 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/__init__.py", line 298, in handler
[Sun Nov 15 16:27:16.261552 2015] [:error] [pid 20785]     return fn_call(inst, *args, **kwargs)
[Sun Nov 15 16:27:16.261558 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/systemcert.py", line 52, in get_transport_cert
[Sun Nov 15 16:27:16.261564 2015] [:error] [pid 20785]     response = self.connection.get(url, self.headers)
[Sun Nov 15 16:27:16.261570 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/client.py", line 115, in get
[Sun Nov 15 16:27:16.261576 2015] [:error] [pid 20785]     data=payload)
[Sun Nov 15 16:27:16.261582 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 319, in get
[Sun Nov 15 16:27:16.261588 2015] [:error] [pid 20785]     return self.request('GET', url, **kwargs)
[Sun Nov 15 16:27:16.261593 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 288, in request
[Sun Nov 15 16:27:16.261600 2015] [:error] [pid 20785]     resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
[Sun Nov 15 16:27:16.261606 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 383, in send
[Sun Nov 15 16:27:16.261612 2015] [:error] [pid 20785]     r = adapter.send(request, **kwargs)
[Sun Nov 15 16:27:16.261617 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 213, in send
[Sun Nov 15 16:27:16.261623 2015] [:error] [pid 20785]     raise SSLError(e)
[Sun Nov 15 16:27:16.261629 2015] [:error] [pid 20785] SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261937 2015] [:error] [pid 20785] ipa: INFO: [jsonserver_session] admin: vaultconfig_show(all=False, raw=False, version=u'2.156'): SSLError
Comment 17 Xiyang Dong 2015-11-17 20:28:59 UTC 
opened https://bugzilla.redhat.com/show_bug.cgi?id=1282935, put bz back to verified
Comment 18 errata-xmlrpc 2015-11-19 12:06:38 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2362.html
Note You need to log in before you can comment on or make changes to this bug.