Bug 1477703

Summary: IPA upgrade fails for latest ipa package
Product: Red Hat Enterprise Linux 7 Reporter: Nikhil Dehadrai <ndehadra>
Component: ipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: ipa-qe <ipa-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.4CC: akasurde, amore, ftweedal, ksiddiqu, ndehadra, pvoborni, rcritten, tscherf
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ipa-4.5.0-21.el7.1.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 16:43:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nikhil Dehadrai 2017-08-02 16:21:30 UTC
Description of problem:
IPA upgrade fails for latest ipa package when upgraded from RHEL 7.1.z to RHEL 7.4.1 (ipa-4.5.0-21.el7_4.1)

Version-Release number of selected component (if applicable):
ipa-4.5.0-21.el7_4.1

How reproducible:
Always

Steps to Reproduce:
1. Setup IPA server on RHEL 7.1.z
2. Configure latest repo to this IPA server.
3. Update this IPA server using command 'yum -y update 'ipa*' sssd'


Actual results:
After step 3, upgrade fails
:: [  BEGIN   ] :: Running 'yum -y update 'ipa*' sssd 'python*''

Cleanup    : nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64     271/271
 
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
HTTPRequestError: Request failed with status 400: Non-2xx response from CA REST API: 400. Invalid profile data
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Verifying  : sssd-krb5-1.15.2-50.el7.x86_64      1/271 

:: [  BEGIN   ] :: Running 'tail -1 /var/log/ipaupgrade.log | grep 'The ipa-server-upgrade command was successful''
:: [   FAIL   ] :: Command 'tail -1 /var/log/ipaupgrade.log | grep 'The ipa-server-upgrade command was successful'' (Expected 0, got 1)


#tail -100 /var/log/ipupgrade.log

2017-08-02T15:02:20Z DEBUG response headers Server: Apache-Coyote/1.1
Cache-Control: private
Expires: Wed, 31 Dec 1969 19:00:00 EST
Set-Cookie: JSESSIONID=F2638D577410A1FB17AE1C20497BA8F1; Path=/ca; Secure; HttpOnly
Content-Type: application/xml
Date: Wed, 02 Aug 2017 15:02:19 GMT

2017-08-02T15:02:20Z DEBUG response body ''
2017-08-02T15:02:20Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2017-08-02T15:02:20Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run
    server.upgrade()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1913, in upgrade
    upgrade_configuration()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1810, in upgrade_configuration
    ca_import_included_profiles(ca)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 396, in ca_import_included_profiles
    return cainstance.import_included_profiles()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1615, in import_included_profiles
    _create_dogtag_profile(profile_id, profile_data, overwrite=True)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1733, in _create_dogtag_profile
    profile_api.update_profile(profile_id, profile_data)
  File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 2079, in update_profile
    body=profile_data
  File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1360, in _ssldo
    % {'status': status, 'explanation': explanation}

2017-08-02T15:02:20Z DEBUG The ipa-server-upgrade command failed, exception: HTTPRequestError: Request failed with status 400: Non-2xx response from CA REST API: 400. Invalid profile data
2017-08-02T15:02:20Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
HTTPRequestError: Request failed with status 400: Non-2xx response from CA REST API: 400. Invalid profile data
2017-08-02T15:02:20Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information


Expected results:
IPA upgrade should be successful

Additional info:
This issue is not observed for IPA server upgrade from:
1. RHEL 7.2.z > RHEL 7.4.1
2. RHEL 7.3.z > RHEL 7.4.1
3. RHEL 7.1.z > RHEL 7.4 (4.5.0.21)

Comment 10 Fraser Tweedale 2017-08-09 05:33:02 UTC
Precise cause of *upgrade* failure is a bug in PKI upgrade scripts:

<<EOF
[root@auto-hv-01-guest03 localhost]# cat /var/log/pki/pki-server-upgrade-10.4.1.log 
Upgrading PKI server configuration at Thu Aug  3 03:46:16 EDT 2017.
Upgrading from version 10.1.2 to 10.1.99:
1. Add TLS Range Support

Upgrading from version 10.1.99 to 10.2.0:
1. Move web application context file
2. Replace Jettison with Jackson
3. Added RESTEasy client
4. Replace RESTEasy application class
5. Remove config path from web.xml

Upgrading from version 10.2.0 to 10.2.1:
No upgrade scriptlets.
Tracker has been set to version 10.2.1.

Upgrading from version 10.2.1 to 10.2.2:
1. Add TLS Range Support

Upgrading from version 10.2.2 to 10.2.3:
1. Move Web application deployment locations
2. Enabled Web application auto deploy
3. Remove dependency on Jackson 2

Upgrading from version 10.2.3 to 10.2.4:
1. Fix instance work folder ownership
2. Fix bindPWPrompt for internalDB

Upgrading from version 10.2.4 to 10.2.5:
1. Add missing OCSP Get Servlet Mapping to upgraded Dogtag 9 instances
2. Fix nuxwdog listener class

Upgrading from version 10.2.5 to 10.2.6:
1. Add new KRA audit events

Upgrading from version 10.2.6 to 10.3.0:
1. Remove inaccessable URLs from server.xml
2. Add Phone Home URLs to TPS section of server.xml.

Upgrading from version 10.3.0 to 10.3.1:
1. Enable Tomcat ALLOW_ENCODED_SLASH parameter
2. Add authz realm constraint and default to registry

Upgrading from version 10.3.1 to 10.3.2:
No upgrade scriptlets.
Tracker has been set to version 10.3.2.

Upgrading from version 10.3.2 to 10.3.3:
No upgrade scriptlets.
Tracker has been set to version 10.3.3.

Upgrading from version 10.3.3 to 10.4.0:
1. Fix JAVA_HOME path
2. Fix server library
3. Fix deployment descriptor
ERROR: [Errno 2] No such file or directory: '/usr/share/pki/server/conf/Catalina/localhost/pki#admin.xml'
Failed upgrading pki-tomcat instance.
Upgrade failed in pki-tomcat: [Errno 2] No such file or directory: '/usr/share/pki/server/conf/Catalina/localhost/pki#admin.xml'

---------------
System migrated
---------------
EOF

A separate ticket should be opened for this against pki-core.

BUT due to possibility of mixed-version topology we also need to address
the fact that the PKI 10.4-only version of the profile is imported when
upgrading from IPA v4.1 or earlier.

Comment 11 Fraser Tweedale 2017-08-09 06:00:02 UTC
Upstream ticket: https://pagure.io/freeipa/issue/7097

Comment 12 Fraser Tweedale 2017-08-09 06:00:38 UTC
Related pki-core BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1479663

Comment 13 Fraser Tweedale 2017-08-10 12:55:38 UTC
Upstream PRs:

-    https://github.com/freeipa/freeipa/pull/964
-    [ipa-4-5] https://github.com/freeipa/freeipa/pull/965

I still need to test these on RHEL explicitly (I have tested it
with ipa v4.1 (f22) upgrade to ipa v4.5 (f26))

Comment 14 Fraser Tweedale 2017-08-11 04:40:22 UTC
Now tested with RHEL 7.1 -> 7.4 upgrade; the fix works.

Comment 24 anuja 2017-12-20 09:21:18 UTC
Marking bz as verified
on 
7.1z > 7.5 Success
7.2z > 7.5 Success
7.3z > 7.5 Success

Please see attachment for console log.

Comment 33 errata-xmlrpc 2018-04-10 16:43:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0918