RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1282935 - ipa upgrade causes vault internal error
Summary: ipa upgrade causes vault internal error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.2
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: Namita Soman
URL:
Whiteboard:
Depends On:
Blocks: 1283883
TreeView+ depends on / blocked
 
Reported: 2015-11-17 20:28 UTC by Xiyang Dong
Modified: 2016-11-04 05:40 UTC (History)
9 users (show)

Fixed In Version: ipa-4.2.0-16.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1283883 (view as bug list)
Environment:
Last Closed: 2016-11-04 05:40:19 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2404 0 normal SHIPPED_LIVE ipa bug fix and enhancement update 2016-11-03 13:56:18 UTC

Description Xiyang Dong 2015-11-17 20:28:12 UTC 
Description of problem:
ipa server upgrade from 4.1.0-18 to 4.2.0-15 causes vault internal error 

Version-Release number of selected component (if applicable):
ipa-server.x86_64 0:4.1.0-18.el7 
ipa-server.x86_64 0:4.2.0-15.el7

How reproducible:
Always

Steps to Reproduce:
1.install 7.1 Master
2.ipa upgrade to newest
3.install kra
4.try vault commands

Actual results:
vault commands prompt out with internal error.

Expected results:
no error occurs

Additional info:

##Master after upgrade## 
[root@mgmt7 ~]# ipa-kra-install -p Secret123 -U

===================================================================
This program will setup Dogtag KRA for the IPA Server.


Configuring KRA server (pki-tomcatd). Estimated time: 2 minutes 6 seconds
  [1/8]: configuring KRA instance
  [2/8]: create KRA agent
  [3/8]: restarting KRA
  [4/8]: configure certmonger for renewals
  [5/8]: configure certificate renewals
  [6/8]: configure HTTP to proxy connections
  [7/8]: add vault container
  [8/8]: apply LDAP updates
Done configuring KRA server (pki-tomcatd).
Restarting the directory server
The ipa-kra-install command was successful

[root@mgmt7 ~]# kinit admin
Password for admin: 

[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: an internal error has occurred

[root@mgmt7 ~]# echo Secret123|base64
U2VjcmV0MTIzCg==

[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred

[root@mgmt7 ~]# grep "KRA is not enabled" /var/log/ipaupgrade.log
2015-11-15T20:18:45Z INFO KRA is not enabled

[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: vault with name "vupgrade" already exists

[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred


From /var/log/httpd/error_log:
.
.
.
[Sun Nov 15 16:27:16.261426 2015] [:error] [pid 20785] ipa: ERROR: non-public: SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261468 2015] [:error] [pid 20785] Traceback (most recent call last):
[Sun Nov 15 16:27:16.261475 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 347, in wsgi_execute
[Sun Nov 15 16:27:16.261502 2015] [:error] [pid 20785]     result = self.Command[name](*args, **options)
[Sun Nov 15 16:27:16.261509 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
[Sun Nov 15 16:27:16.261515 2015] [:error] [pid 20785]     ret = self.run(*args, **options)
[Sun Nov 15 16:27:16.261521 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 760, in run
[Sun Nov 15 16:27:16.261527 2015] [:error] [pid 20785]     return self.execute(*args, **options)
[Sun Nov 15 16:27:16.261533 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/plugins/vault.py", line 1471, in execute
[Sun Nov 15 16:27:16.261539 2015] [:error] [pid 20785]     transport_cert = kra_client.system_certs.get_transport_cert()
[Sun Nov 15 16:27:16.261545 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/__init__.py", line 298, in handler
[Sun Nov 15 16:27:16.261552 2015] [:error] [pid 20785]     return fn_call(inst, *args, **kwargs)
[Sun Nov 15 16:27:16.261558 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/systemcert.py", line 52, in get_transport_cert
[Sun Nov 15 16:27:16.261564 2015] [:error] [pid 20785]     response = self.connection.get(url, self.headers)
[Sun Nov 15 16:27:16.261570 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/client.py", line 115, in get
[Sun Nov 15 16:27:16.261576 2015] [:error] [pid 20785]     data=payload)
[Sun Nov 15 16:27:16.261582 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 319, in get
[Sun Nov 15 16:27:16.261588 2015] [:error] [pid 20785]     return self.request('GET', url, **kwargs)
[Sun Nov 15 16:27:16.261593 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 288, in request
[Sun Nov 15 16:27:16.261600 2015] [:error] [pid 20785]     resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
[Sun Nov 15 16:27:16.261606 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 383, in send
[Sun Nov 15 16:27:16.261612 2015] [:error] [pid 20785]     r = adapter.send(request, **kwargs)
[Sun Nov 15 16:27:16.261617 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 213, in send
[Sun Nov 15 16:27:16.261623 2015] [:error] [pid 20785]     raise SSLError(e)
[Sun Nov 15 16:27:16.261629 2015] [:error] [pid 20785] SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261937 2015] [:error] [pid 20785] ipa: INFO: [jsonserver_session] admin: vaultconfig_show(all=False, raw=False, version=u'2.156'): SSLError
Comment 1 Scott Poore 2015-11-17 20:34:55 UTC 
I saw the same thing testing just now.  I tested 4.2.0-12 which worked but, this issue seems to have appeared after this fix:

https://bugzilla.redhat.com/show_bug.cgi?id=1262996

I did also see the same issue occur with 4.2.0-14.

Here:

https://bugzilla.redhat.com/show_bug.cgi?id=1262996#c2

Endi suggestion in step 1 seems to fix the issue:

Here you see it fail:

[root@rhel7-1 yum.local.d]# ipa-kra-install -p Secret123 -U

===================================================================
This program will setup Dogtag KRA for the IPA Server.


Configuring KRA server (pki-tomcatd). Estimated time: 2 minutes 6 seconds
  [1/8]: configuring KRA instance
  [2/8]: create KRA agent
  [3/8]: restarting KRA
  [4/8]: configure certmonger for renewals
  [5/8]: configure certificate renewals
  [6/8]: configure HTTP to proxy connections
  [7/8]: add vault container
  [8/8]: apply LDAP updates
Done configuring KRA server (pki-tomcatd).
Restarting the directory server
The ipa-kra-install command was successful
[root@rhel7-1 yum.local.d]# kinit admin
Password for admin: 
[root@rhel7-1 yum.local.d]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: an internal error has occurred

And here's the fix:

[root@rhel7-1 httpd]# ls -l /etc/httpd/alias/kra-agent.pem
ls: cannot access /etc/httpd/alias/kra-agent.pem: No such file or directory
[root@rhel7-1 httpd]# pki -d /etc/httpd/alias -C /etc/httpd/alias/pwdfile.txt client-cert-show ipaCert --client-cert /etc/httpd/alias/kra-agent.pem
[root@rhel7-1 httpd]# chown root.apache /etc/httpd/alias/kra-agent.pem
[root@rhel7-1 httpd]# chmod 660 /etc/httpd/alias/kra-agent.pem

and now it seems to work:


[root@rhel7-1 httpd]# ipa vault-add vupgrade2 --type=symmetric --password='mypa55word'
-----------------------
Added vault "vupgrade2"
-----------------------
  Vault name: vupgrade2
  Type: symmetric
  Salt: hepnaMmeogvHi7I/kCEz5w==
  Owner users: admin
  Vault user: admin

[root@rhel7-1 httpd]# echo Secret123|base64
U2VjcmV0MTIzCg==

[root@rhel7-1 httpd]# ipa vault-archive vupgrade2 --password='mypa55word' --data='U2VjcmV0MTIzCg=='
------------------------------------
Archived data into vault "vupgrade2"
------------------------------------

[root@rhel7-1 httpd]# ipa vault-retrieve vupgrade2 --password='mypa55word'
-------------------------------------
Retrieved data from vault "vupgrade2"
-------------------------------------
  Data: U2VjcmV0MTIzCg==
Comment 3 Scott Poore 2015-11-17 20:37:55 UTC 
Note if I remove the kra-agent.pem file and run ipa-server-upgrade, this exports the file for us and resolves the problem as well.

[root@rhel7-1 httpd]# rm /etc/httpd/alias/kra-agent.pem
rm: remove regular file ‘/etc/httpd/alias/kra-agent.pem’? y

[root@rhel7-1 httpd]# ipa-server-upgrade 
Upgrading IPA:
  [1/10]: stopping directory server
  [2/10]: saving configuration
  [3/10]: disabling listeners
  [4/10]: enabling DS global lock
  [5/10]: starting directory server
  [6/10]: updating schema
  [7/10]: upgrading server
  [8/10]: stopping directory server
  [9/10]: restoring configuration
  [10/10]: starting directory server
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
[Verifying that KDC configuration is using ipa-kdb backend]
[Fix DS schema file syntax]
Syntax already fixed
[Removing RA cert from DS NSS database]
RA cert already removed
[Enable sidgen and extdom plugins by default]
[Updating mod_nss protocol versions]
Protocol versions already updated
[Fixing trust flags in /etc/httpd/alias]
Trust flags already processed
[Exporting KRA agent PEM file]
[Removing self-signed CA]
[Checking for deprecated KDC configuration files]
[Checking for deprecated backups of Samba configuration files]
[Setting up Firefox extension]
[Add missing CA DNS records]
IPA CA DNS records already processed
[Removing deprecated DNS configuration options]
[Ensuring minimal number of connections]
[Enabling serial autoincrement in DNS]
[Updating GSSAPI configuration in DNS]
[Updating pid-file configuration in DNS]
Changes to named.conf have been made, restart named
[Upgrading CA schema]
CA schema update complete (no changes)
[Verifying that CA audit signing cert has 2 year validity]
[Update certmonger certificate renewal configuration to version 3]
[Enable PKIX certificate path discovery and validation]
PKIX already enabled
[Authorizing RA Agent to modify profiles]
[Ensuring CA is using LDAPProfileSubsystem]
[Ensuring presence of included profiles]
[Add default CA ACL]
Default CA ACL already added
The IPA services were upgraded
The ipa-server-upgrade command was successful

[root@rhel7-1 httpd]# grep "KRA is not enabled" /var/log/ipaupgrade.log
2015-11-17T19:53:36Z INFO KRA is not enabled

[root@rhel7-1 httpd]# ls -l /etc/httpd/alias/kra-agent.pem
-r--r-----. 1 root apache 3305 Nov 17 14:35 /etc/httpd/alias/kra-agent.pem

[root@rhel7-1 httpd]# ipa vault-retrieve vupgrade2 --password='mypa55word'
-------------------------------------
Retrieved data from vault "vupgrade2"
-------------------------------------
  Data: U2VjcmV0MTIzCg==
Comment 4 Petr Vobornik 2015-11-18 14:34:12 UTC 
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/5462
Comment 5 Jan Cholasta 2015-11-19 10:17:45 UTC 
Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/164fb7b1d19ef316d2ec55a8f85876ccf310544f
ipa-4-2:
https://fedorahosted.org/freeipa/changeset/9d4f383a94b28d415396e1529c747c5e5bbdbc0f
Comment 9 Nikhil Dehadrai 2016-08-08 11:24:49 UTC 
IPA server version: ipa-server-4.4.0-3.el7.x86_64

Verified the bug on the basis of following steps:
1. Verified that no error message is observed during upgrade of IPA server.
2. Verified the bug for upgrade paths:
   a) 7.2(GA) to 7.3
   b) 7.2.z to 7.3 (In my case upgrade from 7.2up6)
3. Refer attachment for console output logs.

Thus on the basis of above observations marking the status of bug to "VERIFIED".
Comment 12 errata-xmlrpc 2016-11-04 05:40:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2404.html
Note You need to log in before you can comment on or make changes to this bug.