RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1456774 - ipa-replica server fails to upgrade
Summary: ipa-replica server fails to upgrade
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base
Version: 7.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: mreynolds
QA Contact: Viktor Ashirov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-30 10:37 UTC by Nikhil Dehadrai
Modified: 2017-08-01 21:17 UTC (History)
9 users (show)

Fixed In Version: 389-ds-base-1.3.6.1-16.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 21:17:44 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2086 0 normal SHIPPED_LIVE 389-ds-base bug fix and enhancement update 2017-08-01 18:37:38 UTC

Description Nikhil Dehadrai 2017-05-30 10:37:43 UTC 
Description of problem:
ipa-replica server fails to upgrade from Rhel 7.1.z to RHEL 7.4.

Version-Release number of selected component (if applicable):
ipa-server-4.5.0-13.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Setup IPA-MASTER server on RHEL 7.1.z.
2. Configure REPLICA server on RHEL 7.1.z against the same IPA-MASTER in Step1.
3. Configure Latest repo links for latest version of IPA-server both on MASTER and REPLICA (selinux packages)
4. Initiate upgrade process for ipa-MASTER server using command:
# yum -y update 'ipa*' sssd 'python*'
5. Initiate upgrade process for ipa-REPLICA server using command:
# yum -y update 'ipa*' sssd 'python*'



Actual results:
1. After step4, IPA-MASTER server is upgraded successfully.
2. After step5, IPA-REPLICA server upgrade fails.

Console output:
---------------
  Cleanup    : libsss_nss_idmap-1.12.2-58.el7_1.18.x86_64 
  Cleanup    : nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64 
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start dirsrv' returned non-zero exit status 1
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information


# tail command to /var/log/ipaupgrade.log:

2017-05-30T09:33:19Z DEBUG stdout=
2017-05-30T09:33:19Z DEBUG stderr=Job for dirsrv failed because the control process exited with error code. See "systemctl status dirsrv" and "journalctl -xe" for details.

2017-05-30T09:33:19Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 95, in __start
    srv.start(self.serverid, ldapi=True)
  File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 157, in start
    instance_name, capture_output=capture_output, wait=wait)
  File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 294, in start
    skip_output=not capture_output)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 495, in run
    raise CalledProcessError(p.returncode, arg_string, str(output))
CalledProcessError: Command '/bin/systemctl start dirsrv' returned non-zero exit status 1

2017-05-30T09:33:19Z DEBUG   [error] CalledProcessError: Command '/bin/systemctl start dirsrv' returned non-zero exit status 1
2017-05-30T09:33:19Z DEBUG   [cleanup]: stopping directory server
2017-05-30T09:33:19Z DEBUG Starting external process
2017-05-30T09:33:19Z DEBUG args=/bin/systemctl stop dirsrv
2017-05-30T09:33:19Z DEBUG Process finished, return code=0
2017-05-30T09:33:19Z DEBUG stdout=
2017-05-30T09:33:19Z DEBUG stderr=
2017-05-30T09:33:19Z DEBUG   duration: 0 seconds
2017-05-30T09:33:19Z DEBUG   [cleanup]: restoring configuration
2017-05-30T09:33:19Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-05-30T09:33:19Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-05-30T09:33:19Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2017-05-30T09:33:19Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-05-30T09:33:19Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-05-30T09:33:19Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2017-05-30T09:33:19Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2017-05-30T09:33:19Z DEBUG   duration: 0 seconds
2017-05-30T09:33:19Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2017-05-30T09:33:19Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run
    server.upgrade()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1888, in upgrade
    data_upgrade.create_instance()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 124, in create_instance
    runtime=90)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 95, in __start
    srv.start(self.serverid, ldapi=True)
  File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 157, in start
    instance_name, capture_output=capture_output, wait=wait)
  File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 294, in start
    skip_output=not capture_output)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 495, in run
    raise CalledProcessError(p.returncode, arg_string, str(output))

2017-05-30T09:33:19Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start dirsrv' returned non-zero exit status 1
2017-05-30T09:33:19Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start dirsrv' returned non-zero exit status 1
2017-05-30T09:33:19Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information


Expected results:
IPA-replica server should upgrade successfully.

Additional info:
This behavior is NOT observed in case of following upgrade paths:
1. RHEL 7.3.z > 7.4
2. RHEL 7.3 > 7.4
3. RHEL 7.2.z > 7.4
Comment 6 Petr Vobornik 2017-05-30 12:10:37 UTC 
DS doesn't start and I see this in DS log:

[30/May/2017:05:33:18 -0400] - slapd stopped.
[30/May/2017:05:33:19.433206886 -0400] - ERR - symload_report_error - Netscape Portable Runtime error -5977: /usr/lib64/dirsrv/plugins/libdes-plugin.so: cannot open shared object file: No such file or directory
[30/May/2017:05:33:19.435264140 -0400] - ERR - symload_report_error - Could not open library "/usr/lib64/dirsrv/plugins/libdes-plugin.so" for plugin DES
[30/May/2017:05:33:19.435992978 -0400] - ERR - slapd_bootstrap_config - The plugin entry [cn=DES,cn=Password Storage Schemes,cn=plugins,cn=config] in the configfile /etc/dirsrv/slapd-TESTRELM-TEST/dse.ldif was invalid. Failed to load plugin's init function.
[30/May/2017:05:33:19.436742880 -0400] - EMERG - main - The configuration files in directory /etc/dirsrv/slapd-TESTRELM-TEST could not be read or were not found.  Please refer to the error log or output for more information.

Thierry could you investigate?
Comment 8 thierry bordaz 2017-05-31 14:07:05 UTC 
The bug is related to the change of default password encryption ciphers from DES to AES (https://pagure.io/389-ds-base/issue/47462).

A new DS plugins (libpbe) replaces the old one (libdse). Update script (ldap/admin/src/scripts/52updateAESplugin.pl) is in charge to replace the plugin name in former 'cn=DES,cn=Password Storage Schemes,cn=plugins,cn=config'.
More specifically changing:

nsslapd-pluginPath: libdes-plugin --> libpbe-plugin

Either this script was not executed or did not updated the entry.

So it is looking as a DS bug during update.
Comment 9 thierry bordaz 2017-05-31 16:13:27 UTC 
Updating dse.ldif with the following change, allows to start DS

dn: cn=DES,cn=Password Storage Schemes,cn=plugins,cn=config
nsslapd-pluginPath: libdes-plugin  --> libpbe-plugin



Note:
cn=AES,cn=Password Storage Schemes,cn=plugins,cn=config is not created in dse.ldif after the upgrade. It is likely a consequence of the previous failure.
Comment 10 thierry bordaz 2017-05-31 17:02:43 UTC 
Looking at the patch

There are one ldif file and a script
ldap/admin/src/scripts/50AES-pbe-plugin.ldif
ldap/admin/src/scripts/52updateAESplugin.pl

the ldif file should create the entry "cn=AES,cn=Password Storage Schemes,cn=plugins,cn=config".

The script is checking that the entry "cn=AES,cn=Password Storage Schemes,cn=plugins,cn=config" exists before updating the DES entry.

In the resulting dse.ldif, the entry "cn=AES,cn=Password Storage Schemes,cn=plugins,cn=config" did not exist. So likely 52updateAESplugin.pl exited before updating DES entry. That would explain why DES pluginpath remained unchanged.

dn: cn=DES,cn=Password Storage Schemes,cn=plugins,cn=config
nsslapd-pluginPath: libdes-plugin 

So far, I can not find trace that 50AES-pbe-plugin.ldif was loaded.
Comment 14 errata-xmlrpc 2017-08-01 21:17:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2086
Note You need to log in before you can comment on or make changes to this bug.