Bug 1282935

Summary: ipa upgrade causes vault internal error
Product: Red Hat Enterprise Linux 7 Reporter: Xiyang Dong <xdong>
Component: ipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: Namita Soman <nsoman>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.2CC: ekeck, jcholast, jkurik, ksiddiqu, mkosek, ndehadra, pvoborni, rcritten, spoore
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: ipa-4.2.0-16.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1283883 (view as bug list) Environment:
Last Closed: 2016-11-04 05:40:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1283883    

Description Xiyang Dong 2015-11-17 20:28:12 UTC 
Description of problem:
ipa server upgrade from 4.1.0-18 to 4.2.0-15 causes vault internal error 

Version-Release number of selected component (if applicable):
ipa-server.x86_64 0:4.1.0-18.el7 
ipa-server.x86_64 0:4.2.0-15.el7

How reproducible:
Always

Steps to Reproduce:
1.install 7.1 Master
2.ipa upgrade to newest
3.install kra
4.try vault commands

Actual results:
vault commands prompt out with internal error.

Expected results:
no error occurs

Additional info:

##Master after upgrade## 
[root@mgmt7 ~]# ipa-kra-install -p Secret123 -U

===================================================================
This program will setup Dogtag KRA for the IPA Server.


Configuring KRA server (pki-tomcatd). Estimated time: 2 minutes 6 seconds
  [1/8]: configuring KRA instance
  [2/8]: create KRA agent
  [3/8]: restarting KRA
  [4/8]: configure certmonger for renewals
  [5/8]: configure certificate renewals
  [6/8]: configure HTTP to proxy connections
  [7/8]: add vault container
  [8/8]: apply LDAP updates
Done configuring KRA server (pki-tomcatd).
Restarting the directory server
The ipa-kra-install command was successful

[root@mgmt7 ~]# kinit admin
Password for admin: 

[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: an internal error has occurred

[root@mgmt7 ~]# echo Secret123|base64
U2VjcmV0MTIzCg==

[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred

[root@mgmt7 ~]# grep "KRA is not enabled" /var/log/ipaupgrade.log
2015-11-15T20:18:45Z INFO KRA is not enabled

[root@mgmt7 ~]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: vault with name "vupgrade" already exists

[root@mgmt7 ~]# ipa vault-archive vupgrade --password='mypa55word' --data='U2VjcmV0MTIzCg=='
ipa: ERROR: an internal error has occurred


From /var/log/httpd/error_log:
.
.
.
[Sun Nov 15 16:27:16.261426 2015] [:error] [pid 20785] ipa: ERROR: non-public: SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261468 2015] [:error] [pid 20785] Traceback (most recent call last):
[Sun Nov 15 16:27:16.261475 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 347, in wsgi_execute
[Sun Nov 15 16:27:16.261502 2015] [:error] [pid 20785]     result = self.Command[name](*args, **options)
[Sun Nov 15 16:27:16.261509 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 443, in __call__
[Sun Nov 15 16:27:16.261515 2015] [:error] [pid 20785]     ret = self.run(*args, **options)
[Sun Nov 15 16:27:16.261521 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 760, in run
[Sun Nov 15 16:27:16.261527 2015] [:error] [pid 20785]     return self.execute(*args, **options)
[Sun Nov 15 16:27:16.261533 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/ipalib/plugins/vault.py", line 1471, in execute
[Sun Nov 15 16:27:16.261539 2015] [:error] [pid 20785]     transport_cert = kra_client.system_certs.get_transport_cert()
[Sun Nov 15 16:27:16.261545 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/__init__.py", line 298, in handler
[Sun Nov 15 16:27:16.261552 2015] [:error] [pid 20785]     return fn_call(inst, *args, **kwargs)
[Sun Nov 15 16:27:16.261558 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/systemcert.py", line 52, in get_transport_cert
[Sun Nov 15 16:27:16.261564 2015] [:error] [pid 20785]     response = self.connection.get(url, self.headers)
[Sun Nov 15 16:27:16.261570 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/pki/client.py", line 115, in get
[Sun Nov 15 16:27:16.261576 2015] [:error] [pid 20785]     data=payload)
[Sun Nov 15 16:27:16.261582 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 319, in get
[Sun Nov 15 16:27:16.261588 2015] [:error] [pid 20785]     return self.request('GET', url, **kwargs)
[Sun Nov 15 16:27:16.261593 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 288, in request
[Sun Nov 15 16:27:16.261600 2015] [:error] [pid 20785]     resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
[Sun Nov 15 16:27:16.261606 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 383, in send
[Sun Nov 15 16:27:16.261612 2015] [:error] [pid 20785]     r = adapter.send(request, **kwargs)
[Sun Nov 15 16:27:16.261617 2015] [:error] [pid 20785]   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 213, in send
[Sun Nov 15 16:27:16.261623 2015] [:error] [pid 20785]     raise SSLError(e)
[Sun Nov 15 16:27:16.261629 2015] [:error] [pid 20785] SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
[Sun Nov 15 16:27:16.261937 2015] [:error] [pid 20785] ipa: INFO: [jsonserver_session] admin: vaultconfig_show(all=False, raw=False, version=u'2.156'): SSLError
Comment 1 Scott Poore 2015-11-17 20:34:55 UTC 
I saw the same thing testing just now.  I tested 4.2.0-12 which worked but, this issue seems to have appeared after this fix:

https://bugzilla.redhat.com/show_bug.cgi?id=1262996

I did also see the same issue occur with 4.2.0-14.

Here:

https://bugzilla.redhat.com/show_bug.cgi?id=1262996#c2

Endi suggestion in step 1 seems to fix the issue:

Here you see it fail:

[root@rhel7-1 yum.local.d]# ipa-kra-install -p Secret123 -U

===================================================================
This program will setup Dogtag KRA for the IPA Server.


Configuring KRA server (pki-tomcatd). Estimated time: 2 minutes 6 seconds
  [1/8]: configuring KRA instance
  [2/8]: create KRA agent
  [3/8]: restarting KRA
  [4/8]: configure certmonger for renewals
  [5/8]: configure certificate renewals
  [6/8]: configure HTTP to proxy connections
  [7/8]: add vault container
  [8/8]: apply LDAP updates
Done configuring KRA server (pki-tomcatd).
Restarting the directory server
The ipa-kra-install command was successful
[root@rhel7-1 yum.local.d]# kinit admin
Password for admin: 
[root@rhel7-1 yum.local.d]# ipa vault-add vupgrade --type=symmetric --password='mypa55word'
ipa: ERROR: an internal error has occurred

And here's the fix:

[root@rhel7-1 httpd]# ls -l /etc/httpd/alias/kra-agent.pem
ls: cannot access /etc/httpd/alias/kra-agent.pem: No such file or directory
[root@rhel7-1 httpd]# pki -d /etc/httpd/alias -C /etc/httpd/alias/pwdfile.txt client-cert-show ipaCert --client-cert /etc/httpd/alias/kra-agent.pem
[root@rhel7-1 httpd]# chown root.apache /etc/httpd/alias/kra-agent.pem
[root@rhel7-1 httpd]# chmod 660 /etc/httpd/alias/kra-agent.pem

and now it seems to work:


[root@rhel7-1 httpd]# ipa vault-add vupgrade2 --type=symmetric --password='mypa55word'
-----------------------
Added vault "vupgrade2"
-----------------------
  Vault name: vupgrade2
  Type: symmetric
  Salt: hepnaMmeogvHi7I/kCEz5w==
  Owner users: admin
  Vault user: admin

[root@rhel7-1 httpd]# echo Secret123|base64
U2VjcmV0MTIzCg==

[root@rhel7-1 httpd]# ipa vault-archive vupgrade2 --password='mypa55word' --data='U2VjcmV0MTIzCg=='
------------------------------------
Archived data into vault "vupgrade2"
------------------------------------

[root@rhel7-1 httpd]# ipa vault-retrieve vupgrade2 --password='mypa55word'
-------------------------------------
Retrieved data from vault "vupgrade2"
-------------------------------------
  Data: U2VjcmV0MTIzCg==
Comment 3 Scott Poore 2015-11-17 20:37:55 UTC 
Note if I remove the kra-agent.pem file and run ipa-server-upgrade, this exports the file for us and resolves the problem as well.

[root@rhel7-1 httpd]# rm /etc/httpd/alias/kra-agent.pem
rm: remove regular file ‘/etc/httpd/alias/kra-agent.pem’? y

[root@rhel7-1 httpd]# ipa-server-upgrade 
Upgrading IPA:
  [1/10]: stopping directory server
  [2/10]: saving configuration
  [3/10]: disabling listeners
  [4/10]: enabling DS global lock
  [5/10]: starting directory server
  [6/10]: updating schema
  [7/10]: upgrading server
  [8/10]: stopping directory server
  [9/10]: restoring configuration
  [10/10]: starting directory server
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
[Verifying that KDC configuration is using ipa-kdb backend]
[Fix DS schema file syntax]
Syntax already fixed
[Removing RA cert from DS NSS database]
RA cert already removed
[Enable sidgen and extdom plugins by default]
[Updating mod_nss protocol versions]
Protocol versions already updated
[Fixing trust flags in /etc/httpd/alias]
Trust flags already processed
[Exporting KRA agent PEM file]
[Removing self-signed CA]
[Checking for deprecated KDC configuration files]
[Checking for deprecated backups of Samba configuration files]
[Setting up Firefox extension]
[Add missing CA DNS records]
IPA CA DNS records already processed
[Removing deprecated DNS configuration options]
[Ensuring minimal number of connections]
[Enabling serial autoincrement in DNS]
[Updating GSSAPI configuration in DNS]
[Updating pid-file configuration in DNS]
Changes to named.conf have been made, restart named
[Upgrading CA schema]
CA schema update complete (no changes)
[Verifying that CA audit signing cert has 2 year validity]
[Update certmonger certificate renewal configuration to version 3]
[Enable PKIX certificate path discovery and validation]
PKIX already enabled
[Authorizing RA Agent to modify profiles]
[Ensuring CA is using LDAPProfileSubsystem]
[Ensuring presence of included profiles]
[Add default CA ACL]
Default CA ACL already added
The IPA services were upgraded
The ipa-server-upgrade command was successful

[root@rhel7-1 httpd]# grep "KRA is not enabled" /var/log/ipaupgrade.log
2015-11-17T19:53:36Z INFO KRA is not enabled

[root@rhel7-1 httpd]# ls -l /etc/httpd/alias/kra-agent.pem
-r--r-----. 1 root apache 3305 Nov 17 14:35 /etc/httpd/alias/kra-agent.pem

[root@rhel7-1 httpd]# ipa vault-retrieve vupgrade2 --password='mypa55word'
-------------------------------------
Retrieved data from vault "vupgrade2"
-------------------------------------
  Data: U2VjcmV0MTIzCg==
Comment 4 Petr Vobornik 2015-11-18 14:34:12 UTC 
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/5462
Comment 5 Jan Cholasta 2015-11-19 10:17:45 UTC 
Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/164fb7b1d19ef316d2ec55a8f85876ccf310544f
ipa-4-2:
https://fedorahosted.org/freeipa/changeset/9d4f383a94b28d415396e1529c747c5e5bbdbc0f
Comment 9 Nikhil Dehadrai 2016-08-08 11:24:49 UTC 
IPA server version: ipa-server-4.4.0-3.el7.x86_64

Verified the bug on the basis of following steps:
1. Verified that no error message is observed during upgrade of IPA server.
2. Verified the bug for upgrade paths:
   a) 7.2(GA) to 7.3
   b) 7.2.z to 7.3 (In my case upgrade from 7.2up6)
3. Refer attachment for console output logs.

Thus on the basis of above observations marking the status of bug to "VERIFIED".
Comment 12 errata-xmlrpc 2016-11-04 05:40:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2404.html