Bug 1282935 - ipa upgrade causes vault internal error
ipa upgrade causes vault internal error
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa (Show other bugs)
7.2
All Linux
urgent Severity urgent
: rc
: ---
Assigned To: IPA Maintainers
Namita Soman
: ZStream
Depends On:
Blocks: 1283883
  Show dependency treegraph
 
Reported: 2015-11-17 15:28 EST by Xiyang Dong
Modified: 2016-11-04 01:40 EDT (History)
9 users (show)

See Also:
Fixed In Version: ipa-4.2.0-16.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1283883 (view as bug list)
Environment:
Last Closed: 2016-11-04 01:40:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Xiyang Dong 2015-11-17 15:28:12 EST
Description of problem:
ipa server upgrade from 4.1.0-18 to 4.2.0-15 causes vault internal error 

Version-Release number of selected component (if applicable):
ipa-server.x86_64 0:4.1.0-18.el7 
ipa-server.x86_64 0:4.2.0-15.el7

How reproducible:
Always

Steps to Reproduce:
1.install 7.1 Master
2.ipa upgrade to newest
3.install kra
4.try vault commands

Actual results:
vault commands prompt out with internal error.

Expected results:
no error occurs

Additional info:

##Master after upgrade## 
[root@mgmt7 ~]# ipa-kra-install -p Secret123 -U

===================================================================
This program will setup Dogtag KRA for the IPA Server.


Configuring KRA server (pki-tomcatd). Estimated time: 2 minutes 6 seconds
  [1/8]: configuring KRA instance
  [2/8]: create KRA agent
  [3/8]: restarting KRA
  [4/8]: configure certmonger for renewals
  [5/8]: configure certificate renewals
  [6/8]: configure HTTP to proxy connections
  [7/8]: add vault container
  [8/8]: apply LDAP updates
Done configuring KRA server (pki-tomcatd).
Restarting the directory server
The ipa-kra-install command was successful

[root@mgmt7 ~]# kinit admin
Password for admin@TESTRELM.TEST: 

[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: an internal error has occurred

[root@mgmt7 ~]# echo Secret123|base64
U2VjcmV0MTIzCg==

[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred

[root@mgmt7 ~]# grep "KRA is not enabled" /var/log/ipaupgrade.log
2015-11-15T20:18:45Z INFO KRA is not enabled

[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: vault with name "vupgrade" already exists

[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred


From /var/log/httpd/error_log:
.
.
.
[Sun Nov 15 16:27:16.261426 2015] [:error] [pid 20785] ipa: ERROR: non-public: SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261468 2015] [:error] [pid 20785] Traceback (most recent call last):
[Sun Nov 15 16:27:16.261475 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 347, in wsgi_execute
[Sun Nov 15 16:27:16.261502 2015] [:error] [pid 20785]     result = self.Command[name](*args, **options)
[Sun Nov 15 16:27:16.261509 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
[Sun Nov 15 16:27:16.261515 2015] [:error] [pid 20785]     ret = self.run(*args, **options)
[Sun Nov 15 16:27:16.261521 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 760, in run
[Sun Nov 15 16:27:16.261527 2015] [:error] [pid 20785]     return self.execute(*args, **options)
[Sun Nov 15 16:27:16.261533 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/plugins/vault.py", line 1471, in execute
[Sun Nov 15 16:27:16.261539 2015] [:error] [pid 20785]     transport_cert = kra_client.system_certs.get_transport_cert()
[Sun Nov 15 16:27:16.261545 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/__init__.py", line 298, in handler
[Sun Nov 15 16:27:16.261552 2015] [:error] [pid 20785]     return fn_call(inst, *args, **kwargs)
[Sun Nov 15 16:27:16.261558 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/systemcert.py", line 52, in get_transport_cert
[Sun Nov 15 16:27:16.261564 2015] [:error] [pid 20785]     response = self.connection.get(url, self.headers)
[Sun Nov 15 16:27:16.261570 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/client.py", line 115, in get
[Sun Nov 15 16:27:16.261576 2015] [:error] [pid 20785]     data=payload)
[Sun Nov 15 16:27:16.261582 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 319, in get
[Sun Nov 15 16:27:16.261588 2015] [:error] [pid 20785]     return self.request('GET', url, **kwargs)
[Sun Nov 15 16:27:16.261593 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 288, in request
[Sun Nov 15 16:27:16.261600 2015] [:error] [pid 20785]     resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
[Sun Nov 15 16:27:16.261606 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 383, in send
[Sun Nov 15 16:27:16.261612 2015] [:error] [pid 20785]     r = adapter.send(request, **kwargs)
[Sun Nov 15 16:27:16.261617 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 213, in send
[Sun Nov 15 16:27:16.261623 2015] [:error] [pid 20785]     raise SSLError(e)
[Sun Nov 15 16:27:16.261629 2015] [:error] [pid 20785] SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261937 2015] [:error] [pid 20785] ipa: INFO: [jsonserver_session] admin@TESTRELM.TEST: vaultconfig_show(all=False, raw=False, version=u'2.156'): SSLError
Comment 1 Scott Poore 2015-11-17 15:34:55 EST
I saw the same thing testing just now.  I tested 4.2.0-12 which worked but, this issue seems to have appeared after this fix:

https://bugzilla.redhat.com/show_bug.cgi?id=1262996

I did also see the same issue occur with 4.2.0-14.

Here:

https://bugzilla.redhat.com/show_bug.cgi?id=1262996#c2

Endi suggestion in step 1 seems to fix the issue:

Here you see it fail:

[root@rhel7-1 yum.local.d]# ipa-kra-install -p Secret123 -U

===================================================================
This program will setup Dogtag KRA for the IPA Server.


Configuring KRA server (pki-tomcatd). Estimated time: 2 minutes 6 seconds
  [1/8]: configuring KRA instance
  [2/8]: create KRA agent
  [3/8]: restarting KRA
  [4/8]: configure certmonger for renewals
  [5/8]: configure certificate renewals
  [6/8]: configure HTTP to proxy connections
  [7/8]: add vault container
  [8/8]: apply LDAP updates
Done configuring KRA server (pki-tomcatd).
Restarting the directory server
The ipa-kra-install command was successful
[root@rhel7-1 yum.local.d]# kinit admin
Password for admin@EXAMPLE.COM: 
[root@rhel7-1 yum.local.d]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: an internal error has occurred

And here's the fix:

[root@rhel7-1 httpd]# ls -l /etc/httpd/alias/kra-agent.pem
ls: cannot access /etc/httpd/alias/kra-agent.pem: No such file or directory
[root@rhel7-1 httpd]# pki -d /etc/httpd/alias -C /etc/httpd/alias/pwdfile.txt client-cert-show ipaCert --client-cert /etc/httpd/alias/kra-agent.pem
[root@rhel7-1 httpd]# chown root.apache /etc/httpd/alias/kra-agent.pem
[root@rhel7-1 httpd]# chmod 660 /etc/httpd/alias/kra-agent.pem

and now it seems to work:


[root@rhel7-1 httpd]# ipa vault-add vupgrade2 --type=symmetric --password='mypa55word'
-----------------------
Added vault "vupgrade2"
-----------------------
  Vault name: vupgrade2
  Type: symmetric
  Salt: hepnaMmeogvHi7I/kCEz5w==
  Owner users: admin
  Vault user: admin

[root@rhel7-1 httpd]# echo Secret123|base64
U2VjcmV0MTIzCg==

[root@rhel7-1 httpd]# ipa vault-archive vupgrade2 --password='mypa55word' --data='U2VjcmV0MTIzCg=='
------------------------------------
Archived data into vault "vupgrade2"
------------------------------------

[root@rhel7-1 httpd]# ipa vault-retrieve vupgrade2 --password='mypa55word'
-------------------------------------
Retrieved data from vault "vupgrade2"
-------------------------------------
  Data: U2VjcmV0MTIzCg==
Comment 3 Scott Poore 2015-11-17 15:37:55 EST
Note if I remove the kra-agent.pem file and run ipa-server-upgrade, this exports the file for us and resolves the problem as well.

[root@rhel7-1 httpd]# rm /etc/httpd/alias/kra-agent.pem
rm: remove regular file ‘/etc/httpd/alias/kra-agent.pem’? y

[root@rhel7-1 httpd]# ipa-server-upgrade 
Upgrading IPA:
  [1/10]: stopping directory server
  [2/10]: saving configuration
  [3/10]: disabling listeners
  [4/10]: enabling DS global lock
  [5/10]: starting directory server
  [6/10]: updating schema
  [7/10]: upgrading server
  [8/10]: stopping directory server
  [9/10]: restoring configuration
  [10/10]: starting directory server
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
[Verifying that KDC configuration is using ipa-kdb backend]
[Fix DS schema file syntax]
Syntax already fixed
[Removing RA cert from DS NSS database]
RA cert already removed
[Enable sidgen and extdom plugins by default]
[Updating mod_nss protocol versions]
Protocol versions already updated
[Fixing trust flags in /etc/httpd/alias]
Trust flags already processed
[Exporting KRA agent PEM file]
[Removing self-signed CA]
[Checking for deprecated KDC configuration files]
[Checking for deprecated backups of Samba configuration files]
[Setting up Firefox extension]
[Add missing CA DNS records]
IPA CA DNS records already processed
[Removing deprecated DNS configuration options]
[Ensuring minimal number of connections]
[Enabling serial autoincrement in DNS]
[Updating GSSAPI configuration in DNS]
[Updating pid-file configuration in DNS]
Changes to named.conf have been made, restart named
[Upgrading CA schema]
CA schema update complete (no changes)
[Verifying that CA audit signing cert has 2 year validity]
[Update certmonger certificate renewal configuration to version 3]
[Enable PKIX certificate path discovery and validation]
PKIX already enabled
[Authorizing RA Agent to modify profiles]
[Ensuring CA is using LDAPProfileSubsystem]
[Ensuring presence of included profiles]
[Add default CA ACL]
Default CA ACL already added
The IPA services were upgraded
The ipa-server-upgrade command was successful

[root@rhel7-1 httpd]# grep "KRA is not enabled" /var/log/ipaupgrade.log
2015-11-17T19:53:36Z INFO KRA is not enabled

[root@rhel7-1 httpd]# ls -l /etc/httpd/alias/kra-agent.pem
-r--r-----. 1 root apache 3305 Nov 17 14:35 /etc/httpd/alias/kra-agent.pem

[root@rhel7-1 httpd]# ipa vault-retrieve vupgrade2 --password='mypa55word'
-------------------------------------
Retrieved data from vault "vupgrade2"
-------------------------------------
  Data: U2VjcmV0MTIzCg==
Comment 4 Petr Vobornik 2015-11-18 09:34:12 EST
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/5462
Comment 9 Nikhil Dehadrai 2016-08-08 07:24:49 EDT
IPA server version: ipa-server-4.4.0-3.el7.x86_64

Verified the bug on the basis of following steps:
1. Verified that no error message is observed during upgrade of IPA server.
2. Verified the bug for upgrade paths:
   a) 7.2(GA) to 7.3
   b) 7.2.z to 7.3 (In my case upgrade from 7.2up6)
3. Refer attachment for console output logs.

Thus on the basis of above observations marking the status of bug to "VERIFIED".
Comment 12 errata-xmlrpc 2016-11-04 01:40:19 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2404.html

Note You need to log in before you can comment on or make changes to this bug.