Bug 1365572 - IPA server broken after upgrade
Summary: IPA server broken after upgrade
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.3
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: Kaleem
: 1365507 (view as bug list)
Depends On: 1364071
Blocks: 1286635
TreeView+ depends on / blocked
Reported: 2016-08-09 15:00 UTC by Nikhil Dehadrai
Modified: 2016-11-04 06:00 UTC (History)
4 users (show)

Fixed In Version: ipa-4.4.0-9.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2016-11-04 06:00:44 UTC
Target Upstream Version:

Attachments (Terms of Use)
Console Output log (25.69 KB, text/plain)
2016-08-10 09:05 UTC, Nikhil Dehadrai
no flags Details

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2404 0 normal SHIPPED_LIVE ipa bug fix and enhancement update 2016-11-03 13:56:18 UTC

Description Nikhil Dehadrai 2016-08-09 15:00:27 UTC
Description of problem:
IPA server broken after upgrade from 7.1 to 7.3

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Setup IPA server on RHEL 7.1 (In my case ipa-server-4.1.0-18.el7.x86_64)
2. Setup repo links for RHEL 7.3
3. Upgrade IPA server using command "yum -y update'ipa*' sssd"
4. After upgrade try accessing server UI.
5. After Upgrade try to restart IPA service using command "ipactl restart"
6. After upgrade check for ipaupgrade.log at /var/log path

Actual results:
1. After step4, Server UI is not accessible.
2. After step5, ipactl service fails:
   #[root@auto-hv-01-guest09 ~]# ipactl restart
    Unexpected error
    ImportError: No module named packages.urllib3.exceptions
3. After step6, noticed that ipaupgrade.log file is present but log is not populated.
   [root@auto-hv-01-guest09 log]# cat /var/log/ipaupgrade.log
   [root@auto-hv-01-guest09 log]# ls -l /var/log/ipaupgrade.log
   -rw-r--r--. 1 root root 0 Aug  9 09:09 /var/log/ipaupgrade.log
4. Also noticed following errors in /var/log/httpd/error.log file
   ipa: ERROR: cannot connect to 'https://auto-hv-01-guest09.testrelm.test/ipa/session/json': Internal Server Error
[root@auto-hv-01-guest09 ~]# tail -f /var/log/httpd/error_log 
[Tue Aug 09 09:43:17.417652 2016] [:error] [pid 20091] ipa: INFO: [jsonserver_session] admin@TESTRELM.TEST: env((u'api_version',), version=u'2.0'): SUCCESS
[Tue Aug 09 09:43:17.480375 2016] [:error] [pid 20092] ipa: INFO: [jsonserver_session] admin@TESTRELM.TEST: schema: CommandError
[Tue Aug 09 09:43:17.543136 2016] [:error] [pid 20091] ipa: INFO: [jsonserver_session] admin@TESTRELM.TEST: env((u'api_version',), version=u'2.0'): SUCCESS
[Tue Aug 09 09:43:17.601016 2016] [:error] [pid 20081] AH00000: sd_notifyf returned an error -111
[Tue Aug 09 09:43:17.696469 2016] [:error] [pid 20092] ipa: INFO: [jsonserver_session] admin@TESTRELM.TEST: schema: CommandError
[Tue Aug 09 09:43:19.183124 2016] [:error] [pid 17285] ipa: ERROR: Failed to start IPA: No module named packages.urllib3.exceptions
[Tue Aug 09 09:43:19.191890 2016] [:error] [pid 17284] ipa: ERROR: Failed to start IPA: No module named packages.urllib3.exceptions

Expected results:
Upgrade should be successful with no errors and having accessible server UI. 
Respective upgrade log should be updated correctly.
IPA service should be restarted successfully

Additional Information:
This issue was not noticed for upgrade from 7.2 to 7.3. (In my case 7.2GA to 7.3)

Comment 3 Petr Vobornik 2016-08-09 16:58:24 UTC
Seems to me as duplicate of bug 1364071. Nikhil, could you retest with pki-core-10.3.3-5.el7

Comment 4 Nikhil Dehadrai 2016-08-10 09:05:21 UTC
Hi Petr,

After updating the pki package, I am still seeing the same issue. Another thing I noticed is some errors related to "sd_notify" under httpd/error_log file.

I am attaching the console output log for reference.

Let me know if you need anymore details.

Comment 5 Nikhil Dehadrai 2016-08-10 09:05:57 UTC
Created attachment 1189515 [details]
Console Output log

Console Output log

Comment 6 Petr Vobornik 2016-08-17 17:03:44 UTC
Nikhil, could you paste output of:

rpm -qa python-urllib3 python-requests

Comment 13 Petr Vobornik 2016-08-23 12:06:53 UTC
pki-core spec needs to be raise according to comment 12

Comment 14 Petr Vobornik 2016-08-23 12:10:14 UTC
*** Bug 1365507 has been marked as a duplicate of this bug. ***

Comment 15 Petr Vobornik 2016-08-24 07:21:07 UTC
pki build with the spec change exists, so IPA should raise requires to: pki-core-10.3.3-7.el7

Moving to POST to indicate that no patch is needed.

Comment 17 Nikhil Dehadrai 2016-09-07 09:04:23 UTC
IPA Server version: ipa-server-4.4.0-9.el7.x86_64

Verified the bug bug on the basis of following points:
1. Verified that upgrade of IPA server setup on RHEL 7.1 to RHEL 7.3 is successful.
2. Verified that the error messages are observed inside "/var/log/ipaupgrade.log".
3. Verified that no error messages are observed inside "/var/log/httpd/error_log"
4. Verified that the server UI is accessible after the upgrade.
5. Verified that services can be run for "ipactl restart" and "ipactl status" successfully.
6. Also verified that "kinit admin" command runs successfully after the upgrade.

Thus on the basis of above observations marking the status of bug to "VERIFIED".

Comment 20 errata-xmlrpc 2016-11-04 06:00:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.