Bug 1364510

Summary: ipa dirsrv will not start after upgrade
Product: Red Hat Enterprise Linux 7 Reporter: Scott Poore <spoore>
Component: 389-ds-baseAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED DUPLICATE QA Contact: Viktor Ashirov <vashirov>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: nkinder, pvoborni, rcritten, rmeggins, spoore
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-10 23:48:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ipaupgrade.log with failure
none
dirsrv errors log none

Description Scott Poore 2016-08-05 14:23:04 UTC
Description of problem:

After minor release update, ipa won't start and shows dirsrv error.  This is seen during yum update itself:

IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start dirsrv' returned non-zero exit status 1
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Then I tried a restart after upgrade and was told to re-run ipa-server-upgrade:

[root@rhel7-5 yum.local.d]# ipactl restart
Upgrade required: please run ipa-server-upgrade command
Aborting ipactl

[root@rhel7-5 yum.local.d]# ipa-server-upgrade
Upgrading IPA:
  [1/8]: saving configuration
  [2/8]: disabling listeners
  [3/8]: enabling DS global lock
  [4/8]: starting directory server
  [error] CalledProcessError: Command '/bin/systemctl start dirsrv' returned non-zero exit status 1
  [cleanup]: stopping directory server
  [cleanup]: restoring configuration
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start dirsrv' returned non-zero exit status 1
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Version-Release number of selected component (if applicable):
Before upgrade:
389-ds-base-1.3.5.10-5.el7.x86_64
ipa-server-4.4.0-4.el7.x86_64

After upgrade:
389-ds-base-1.3.5.10-6.el7.x86_64
ipa-server-4.4.0-5.el7.x86_64


How reproducible:
Unknown

Steps to Reproduce:
1.  Install IPA server with "Before upgrade" versions listed
2.  point to repo with new rpms
3.  yum -y update

Actual results:
cannot start IPA

Expected results:
IPA/dirsrv starts

Additional info:

Comment 1 Scott Poore 2016-08-05 14:25:34 UTC
Created attachment 1187931 [details]
ipaupgrade.log with failure

Comment 2 Scott Poore 2016-08-05 14:27:07 UTC
Created attachment 1187933 [details]
dirsrv errors log

Comment 3 Petr Vobornik 2016-08-05 14:31:34 UTC
Another DS crash today... Could it be a dup of following bugs?

* bug 1364452
* bug 1316580
* bug 1364377

Comment 5 Scott Poore 2016-08-05 14:40:04 UTC
Petr,

Yes, this bug here looks like the first two bugs you listed.  Not sure about the third.

From /var/log/messages:

Aug  5 09:30:06 rhel7-5 systemd: Starting 389 Directory Server TESTRELM-TEST....
Aug  5 09:30:06 rhel7-5 systemd: Failed at step EXEC spawning /usr/sbin/ds_systemd_ask_password_acl: No such file or directory
Aug  5 09:30:06 rhel7-5 systemd: dirsrv: control process exited, code=exited status=203
Aug  5 09:30:06 rhel7-5 systemd: Failed to start 389 Directory Server TESTRELM-TEST..


So we're missing /usr/sbin/ds_systemd_ask_password_acl

I'm not quite sure which one to mark this one as a dup of or leave it for now and we'll see what DS guys say about bug #1316580 issue.

Should I re-assign this to 389-ds-base?

Thanks,
Scott

Comment 6 Kaleem 2016-08-08 06:20:06 UTC
(In reply to Scott Poore from comment #5)
> Petr,
> 
> Yes, this bug here looks like the first two bugs you listed.  Not sure about
> the third.
> 
> From /var/log/messages:
> 
> Aug  5 09:30:06 rhel7-5 systemd: Starting 389 Directory Server
> TESTRELM-TEST....
> Aug  5 09:30:06 rhel7-5 systemd: Failed at step EXEC spawning
> /usr/sbin/ds_systemd_ask_password_acl: No such file or directory
> Aug  5 09:30:06 rhel7-5 systemd: dirsrv: control
> process exited, code=exited status=203
> Aug  5 09:30:06 rhel7-5 systemd: Failed to start 389 Directory Server
> TESTRELM-TEST..
> 
> 
> So we're missing /usr/sbin/ds_systemd_ask_password_acl
> 
> I'm not quite sure which one to mark this one as a dup of or leave it for
> now and we'll see what DS guys say about bug #1316580 issue.
I reported another bug https://bugzilla.redhat.com/show_bug.cgi?id=1364452#c4 which is duplicate of 1316580. 
And this looks too duplicate of #1316580 .
> 
> Should I re-assign this to 389-ds-base?
> 
> Thanks,
> Scott

Comment 7 Petr Vobornik 2016-08-08 08:25:20 UTC
Moving to dirsrv according to comments 4-5.

Comment 8 Noriko Hosoi 2016-08-10 19:46:46 UTC
Hi Scott,

Did you have a chance to try 389-ds-base-1.3.5.10-7.el7?
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=508177

If you could update this bug with the result, we'd appreciate it.

Sorry about the regression.

Comment 9 Scott Poore 2016-08-10 21:59:26 UTC
Hi Noriko,

Yes, I had upgraded earlier today but, to be sure, I did one for this with the versions in question.   Looks like this is good now.  Do you want me to mark this one verified or close as dup of one of the others?

Before upgrade:

# rpm -q ipa-server 389-ds-base
ipa-server-4.4.0-4.el7.x86_64
389-ds-base-1.3.5.10-5.el7.x86_64


# ipa-server-install --setup-dns --forwarder=192.168.122.1 --reverse-zone=122.168.192.in-addr.arpa. --allow-zone-overlap -r EXAMPLE.COM -a Secret123 -p Secret123 -U
...

install looked normal.

pointed to repo with newer rpms and upgraded.

after upgrade:

# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

# ipactl restart
Stopping pki-tomcatd Service
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting ipa_memcached Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful

# rpm -q ipa-server 389-ds-base
ipa-server-4.4.0-7.el7.x86_64
389-ds-base-1.3.5.10-7.el7.x86_64


Thanks,
Scott

Comment 10 Noriko Hosoi 2016-08-10 23:33:27 UTC
Thank you soooo much, Scott!  

Yes, 389-ds-base-1.3.5.10-6.el7 was broken with a regression.

Since this bug is not acked yet and it is a regression introduced by bug 1316580 which is being implemented for rhel-7.3, could you please close this bug as a duplicate of bug 1316580?

Thanks!
--noriko

Comment 11 Scott Poore 2016-08-10 23:48:46 UTC
Sure thing.  I'll close this one as dup of that.

*** This bug has been marked as a duplicate of bug 1316580 ***