Bug 1949173 - Not able to start dirsrv during IPA upgrade
Summary: Not able to start dirsrv during IPA upgrade
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.9
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: rc
: ---
Assignee: Florence Blanc-Renaud
QA Contact: ipa-qe
URL:
Whiteboard: sync-to-jira
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-13 15:42 UTC by Maria
Modified: 2023-08-14 08:27 UTC (History)
20 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FREEIPA-7817 0 None None None 2022-02-07 16:21:12 UTC
Red Hat Knowledge Base (Solution) 6814721 0 None None None 2022-03-17 05:52:53 UTC
Red Hat Knowledge Base (Solution) 6817171 0 None None None 2022-03-16 17:05:51 UTC

Description Maria 2021-04-13 15:42:03 UTC
Description of problem:
   IPA is not starting

# ipactl start -d
IPA version error: data needs to be upgraded (expected version '4.6.8-5.el7_9.4', current version '4.6.8-5.el7')
Automatically running upgrade, for details see /var/log/ipaupgrade.log
Be patient, this may take a few minutes.
ipa: DEBUG: Starting external process
ipa: DEBUG: args=/usr/sbin/ipa-server-upgrade
ipa: DEBUG: Process finished, return code=1
ipa: DEBUG: stdout=Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/9]: saving configuration
  [2/9]: disabling listeners
  [3/9]: enabling DS global lock
  [4/9]: disabling Schema Compat
  [5/9]: starting directory server
  [error] CalledProcessError: Command '/bin/systemctl start dirsrv' returned non-zero exit status 1
  [cleanup]: stopping directory server
  [cleanup]: restoring configuration

ipa: DEBUG: stderr=IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start dirsrv' returned non-zero exit status 1
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Automatic upgrade failed: IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start dirsrv' returned non-zero exit status 1
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

See the upgrade log for more details and/or run /usr/sbin/ipa-server-upgrade again
ipa: DEBUG:   File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 1015, in run_script
    return_value = main_function()

  File "/sbin/ipactl", line 598, in main
    ipa_start(options)

  File "/sbin/ipactl", line 288, in ipa_start
    version_check()

  File "/sbin/ipactl", line 161, in version_check
    raise IpactlError("Aborting ipactl")

ipa: DEBUG: The ipactl command failed, exception: IpactlError: Aborting ipactl
Aborting ipactl


Version-Release number of selected component (if applicable):
   OS Version: Server 7.9 (Maipo)
   ipa-server-4.6.8-5.el7_9.4.x86_64
   389-ds-base-1.3.10.2-10.el7_9.x86_64

How reproducible:
   The customer started facing this issue after upgrading IPA packages

Steps to Reproduce:
   1. the customer updated to RHEL 7.9 
   2. after, when trying to start IPA, it failed with the above error


Actual results:
   ipa: DEBUG: The ipactl command failed, exception: IpactlError: Aborting ipactl
   Aborting ipactl

Expected results:
   IPA services should successfuly start

Additional info:
   the work around was to delete files /var/lib/sss/pubconf/krb5.include.d/

Comment 4 Rob Crittenden 2021-04-13 15:55:23 UTC
I'm going to assign this to the sssd package since it owns these files.

Comment 25 Ding-Yi Chen 2022-03-15 00:07:58 UTC
To sum up:

1. dirsrv not started is caused by SSSD krb5 locator temporary file
   a. Bug 1952569 address this
   b. sssd-2.6.1-1.el8 in RHEL 8.6 fixed it
2. Workaround for customer that have not get above fix:
   Moving SSSD krb locator temporary file to other location, e.g.
   mv /var/lib/sss/pubconf/krb5.include.d/krb5_libdefaultsXXXXXX /tmp

Comment 26 Runar Lundgren 2022-03-15 04:43:01 UTC
The KCS "'ipactl start' fails to start Directory Server" (https://access.redhat.com/solutions/6814721) has been written with the problem description, solution  and workaround.

Comment 27 Florence Blanc-Renaud 2022-03-15 07:04:16 UTC
The root of the issue is not the presence of the SSSD krb locator temporary file, but the *permissions* of the temporary file.

When IdM is configured, /etc/krb5.conf contains:
----- 8< -----
includedir /etc/krb5.conf.d/
includedir /var/lib/sss/pubconf/krb5.include.d/
...
----- >8 -----

As a consequence, kerberos client applications try to read all the files below /etc/krb5.conf.d/ and /var/lib/sss/pubconf/krb5.include.d/ and if one of them is not readable by the app, the initialization of the kerberos context fails.

The diagnostic can be done based on the content of dirsrv error log:

[root@server ~]# grep krb5_init_context /var/log/dirsrv/slapd-IPA-TEST/errors
[15/Mar/2022:06:56:49.822456662 +0000] - ERR - ipalockout_get_global_config - [file ipa_lockout.c, line 178]: krb5_init_context failed (-1429577697)
[15/Mar/2022:06:56:49.828182116 +0000] - ERR - ipaenrollment_start - [file ipa_enrollment.c, line 398]: krb5_init_context failed
[15/Mar/2022:06:56:49.834718739 +0000] - ERR - ipapwd_start - [file ipa_pwd_extop.c, line 1879]: krb5_init_context failed

Can the KCS be updated with the above information?

Comment 28 Ding-Yi Chen 2022-03-15 07:47:27 UTC
(In reply to Florence Blanc-Renaud from comment #27)

> Can the KCS be updated with the above information?

I have made the required modification. Please review

Comment 29 Florence Blanc-Renaud 2022-03-15 08:50:55 UTC
Thanks or the update, LGTM.

Comment 30 Jeremy Absher 2022-03-15 14:53:59 UTC
A KCS has been created for this BZ: https://access.redhat.com/solutions/6817171


Note You need to log in before you can comment on or make changes to this bug.