Bug 1606541 - freeipa-server missing dependency on 389-ds-base-legacy-tools, results in "No such file or directory: '/usr/sbin/setup-ds.pl'" error during ipa-server-install with 389-ds-base 1.4.0.12+
Summary: freeipa-server missing dependency on 389-ds-base-legacy-tools, results in "No...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: freeipa
Version: 28
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: IPA Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-20 20:31 UTC by Lukas Slebodnik
Modified: 2018-08-19 02:25 UTC (History)
11 users (show)

Fixed In Version: freeipa-4.7.0-1.fc28
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-08-19 02:25:56 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1607635 None CLOSED Segfault in ldif2db during ipa-server-install on current Rawhide 2019-06-28 00:19:10 UTC

Internal Links: 1607635

Description Lukas Slebodnik 2018-07-20 20:31:47 UTC
Description of problem:
The installation of freeIPA server fails on fedora 28 with enabed repo updates-testing.

Version-Release number of selected component (if applicable):
freeipa-server-4.6.90.pre2-3.fc28.x86_64
389-ds-base-1.4.0.12-1.fc28.x86_64

How reproducible:
Deterministic

Steps to Reproduce:
1. dnf -y install --enablerepo=updates-testing freeipa-server-dns
2. /usr/sbin/ipa-server-install --hostname=host.testrelm.test -r TESTRELM.TEST -n testrelm.test -p Secret123 -a Secret123

Actual results:
//snip

Synchronizing time
Using default chrony configuration.
Time synchronization was successful.
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/44]: creating directory server instance
  [error] FileNotFoundError: [Errno 2] No such file or directory: '/usr/sbin/setup-ds.pl': '/usr/sbin/setup-ds.pl'
ipapython.admintool: ERROR    [Errno 2] No such file or directory: '/usr/sbin/setup-ds.pl': '/usr/sbin/setup-ds.pl'
ipapython.admintool: ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

Expected results:
Installation pass without any failure

Additional info:

sh# dnf --enablerepo=updates-testing provides /usr/sbin/setup-ds.pl
Last metadata expiration check: 0:00:46 ago on Fri Jul 20 20:30:24 2018.
389-ds-base-legacy-tools-1.4.0.12-1.fc28.x86_64 : Legacy utilities for 389
                                                : Directory Server (%{variant})
Repo        : updates-testing
Matched from:
Filename    : /usr/sbin/setup-ds.pl

389-ds-base-legacy-tools-1.4.0.11-2.fc28.x86_64 : Legacy utilities for 389
                                                : Directory Server (%{variant})
Repo        : updates
Matched from:
Filename    : /usr/sbin/setup-ds.pl

389-ds-base-1.4.0.6-2.fc28.x86_64 : 389 Directory Server (base)
Repo        : fedora
Matched from:
Filename    : /usr/sbin/setup-ds.pl

Comment 1 Jan Pazdziora 2018-07-23 11:15:33 UTC
Even if I explicitly add /usr/sbin/setup-ds.pl to the dnf operation, on rawhide things fail with

 [error] RuntimeError: failed to create DS instance CalledProcessError(Command ['/usr/sbin/setup-ds.pl', '--silent', '--logfile', '-', '-f', '/tmp/tmpls_vesa4'] returned non-zero exit status 1: '')
FreeIPA server configuration failed.

https://travis-ci.org/adelton/freeipa-container/jobs/407095858

Comment 2 mreynolds 2018-07-23 11:37:48 UTC
Correct we now use "dscreate" instead of setup-ds.pl in 1.4.0. 

Or, you can install "389-ds-base-legacy-tools" package to get the old perl scripts including setup-ds.pl.  This change has been in the works for a long time - it has been part of the development plan for 1.4.0 from the very beginning.  IPA has been aware of this change, but you can still install the legacy tools package if you want to use the old perl installer.

Comment 3 Alexander Bokovoy 2018-07-23 11:39:07 UTC
Mark, look at the comment 1: installing 389-ds-base-legacy-tools aren't fixing the problem.

Comment 4 Alexander Bokovoy 2018-07-23 11:43:44 UTC
... and it would really have been good to coordinate with FreeIPA when the old method is not available for use by default. Pull request to switch to dscreate is still being reviewed, also pending changes in 389-ds side.

Comment 5 Jan Pazdziora 2018-07-23 12:16:08 UTC
For the record, ipaserver-install.log has the following in it:

[23/Jul/2018:12:14:25.556762318 +0000] - INFO - ldbm_ancestorid_new_idl_create_index - import userRoot: Created ancestorid index (new idl).
[23/Jul/2018:12:14:25.565269653 +0000] - INFO - import_main_offline - import userRoot: Flushing caches...
[23/Jul/2018:12:14:25.573994410 +0000] - INFO - import_main_offline - import userRoot: Closing files...
[23/Jul/2018:12:14:25.645427122 +0000] - INFO - dblayer_pre_close - All database threads now stopped
[23/Jul/2018:12:14:25.647679208 +0000] - INFO - import_main_offline - import userRoot: Import complete.  Processed 1 entries in 1 seconds. (1.00 entries/sec)
/usr/sbin/ldif2db: line 116:   111 Segmentation fault      /usr/sbin/ns-slapd ldif2db -D /etc/dirsrv/slapd-EXAMPLE-TEST -n "userRoot" -i "/var/lib/dirsrv/boot.ldif"

[18/07/23:12:14:25] - [Setup] Fatal Error: Could not create directory server instance 'EXAMPLE-TEST'.
Error: Could not create directory server instance 'EXAMPLE-TEST'.
[18/07/23:12:14:25] - [Setup] Fatal Exiting . . .
Log file is '-'

Exiting . . .
Log file is '-'


2018-07-23T12:14:25Z DEBUG stderr=
2018-07-23T12:14:25Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/ipaserver/install/dsinstance.py", line 571, in __create_instance
    ipautil.run(args)
  File "/usr/lib/python3.7/site-packages/ipapython/ipautil.py", line 572, in run
    p.returncode, arg_string, output_log, error_log

Comment 6 mreynolds 2018-07-23 12:22:28 UTC
Jan, this from running setup-ds.pl right?   Not sure why ldif2db would be crashing since this area of the code has not been touched, either way that would need to be fixed. 

Is there a core file to look at by any chance?

Comment 7 mreynolds 2018-07-23 12:31:16 UTC
(In reply to Alexander Bokovoy from comment #4)
> ... and it would really have been good to coordinate with FreeIPA when the
> old method is not available for use by default. Pull request to switch to
> dscreate is still being reviewed, also pending changes in 389-ds side.

I'm sorry that this causing issues, but FreeIPA has known for a very long time that we were making this change.   There are also no outstanding issues with dscreate in DS - I think you are referring to the python openssl requirement.  That was also addressed several releases ago which was communicated with the FreeIPA team.

Comment 8 Jan Pazdziora 2018-07-23 12:35:00 UTC
(In reply to mreynolds from comment #6)
> Jan, this from running setup-ds.pl right?

It's from running ipa-server-install. My understanding it is calls setup-ds.pl.

> Is there a core file to look at by any chance?

No. This is in container, cores are hard to get by.

Comment 9 Alexander Bokovoy 2018-07-23 12:37:38 UTC
Mark, the issue is that you are pushing a change to Fedora that is not coordinated on package level with freeipa, so you are guaranteed to break freeipa installation. For such cases Fedora has joint bodhi requests that can include multiple packages. This is what matters to Fedora, specifically.

FreeIPA 4.7 does not include the patches to support dscreate yet as they are not yet ready (review-wise). Thus, Fedora packages need to use ds-setup.pl and knowing about that, it is better to recall your bodhi update and fix dependencies.

It is all about coordination: removing dependencies without making sure they will be picked up by another package is a guaranteed way to kill deployments. Let's remove the dependency when it is known to be not needed.

Comment 10 mreynolds 2018-07-23 14:26:11 UTC
(In reply to Alexander Bokovoy from comment #9)
> Mark, the issue is that you are pushing a change to Fedora that is not
> coordinated on package level with freeipa, so you are guaranteed to break
> freeipa installation. For such cases Fedora has joint bodhi requests that
> can include multiple packages. This is what matters to Fedora, specifically.
> 
> FreeIPA 4.7 does not include the patches to support dscreate yet as they are
> not yet ready (review-wise). Thus, Fedora packages need to use ds-setup.pl
> and knowing about that, it is better to recall your bodhi update and fix
> dependencies.
> 
> It is all about coordination: removing dependencies without making sure they
> will be picked up by another package is a guaranteed way to kill
> deployments. Let's remove the dependency when it is known to be not needed.

I said I was sorry, I was not being sarcastic.  This one slipped through the cracks - especially since we still provide setup-ds.pl in 389-ds-base-legacy-tools package.  That package could easily be made a requirement in the IPA spec file - is that an option or do you actually need me to revert the patch and rebuild DS?  Or, can I help review the dscreate PR in FreeIPA?  That PR should be a priority in FreeIPA anyway.

Comment 11 mreynolds 2018-07-23 14:27:44 UTC
(In reply to Alexander Bokovoy from comment #3)
> Mark, look at the comment 1: installing 389-ds-base-legacy-tools aren't
> fixing the problem.

setup-ds.pl from the legacy tools package works for me.  If there a reproducible test case then please file a new bug.

Comment 12 Alexander Bokovoy 2018-07-23 14:31:04 UTC
(In reply to mreynolds from comment #10)
> I said I was sorry, I was not being sarcastic.  This one slipped through the
> cracks - especially since we still provide setup-ds.pl in
> 389-ds-base-legacy-tools package.  That package could easily be made a
> requirement in the IPA spec file - is that an option or do you actually need
> me to revert the patch and rebuild DS?  Or, can I help review the dscreate
> PR in FreeIPA?  That PR should be a priority in FreeIPA anyway.
Thanks.

Thomas W. is working right now on FreeIPA 4.7.0 packages for rawhide and F28, they will include the Requires: 389-ds-base-legacy-tools there. Hopefully, this would happen today/tomorrow. I'd appreciate if we could add freeipa package to the same updates bodhi request in F28 so that people don't get affected by the change (aside from updates-testing right now).

As for the PR, there few changes that need to be done there to complete review as 389-ds now includes all required fixes to avoid the double logging. Once these are done, we can push the PR upstream. However, it is not going to be part of 4.7.0 as the release has already been done.

Comment 13 mreynolds 2018-07-23 14:37:29 UTC
(In reply to mreynolds from comment #11)
> (In reply to Alexander Bokovoy from comment #3)
> > Mark, look at the comment 1: installing 389-ds-base-legacy-tools aren't
> > fixing the problem.
> 
> setup-ds.pl from the legacy tools package works for me.  If there a
> reproducible test case then please file a new bug.

FYI, the crash that was reported in comment 1 is not related to setup-ds.pl, but actually the ldif2db utility.  So it is unrelated to this change, and will probably still be a problem after I revert the legacy tool package.  It needs more investigation...

Comment 14 mreynolds 2018-07-23 14:41:36 UTC
(In reply to Alexander Bokovoy from comment #12)
> (In reply to mreynolds from comment #10)
> > I said I was sorry, I was not being sarcastic.  This one slipped through the
> > cracks - especially since we still provide setup-ds.pl in
> > 389-ds-base-legacy-tools package.  That package could easily be made a
> > requirement in the IPA spec file - is that an option or do you actually need
> > me to revert the patch and rebuild DS?  Or, can I help review the dscreate
> > PR in FreeIPA?  That PR should be a priority in FreeIPA anyway.
> Thanks.
> 
> Thomas W. is working right now on FreeIPA 4.7.0 packages for rawhide and
> F28, they will include the Requires: 389-ds-base-legacy-tools there.
> Hopefully, this would happen today/tomorrow. I'd appreciate if we could add
> freeipa package to the same updates bodhi request in F28 so that people
> don't get affected by the change (aside from updates-testing right now).
> 
> As for the PR, there few changes that need to be done there to complete
> review as 389-ds now includes all required fixes to avoid the double
> logging. 

Double logging, is this a new bug or a old one?  We are seeing "double logging" in lib389 cli tools after we very recently got a contribution from a community member (not fixed yet).  Just checking if it's the same thing, or we still have a blocker for ipa.

Comment 15 Alexander Bokovoy 2018-07-23 14:53:17 UTC
I haven't looked into details of the double logging, this is one of todo items for the PR: https://github.com/freeipa/freeipa/pull/1563#issuecomment-402403691

Comment 16 mreynolds 2018-07-23 15:02:31 UTC
Also, I just installed freeipa 4.6.90 on F28 with legacy tools and it succeeded:

Synchronizing time
No SRV records of NTP servers found and no NTP server or pool address was provided.
Using default chrony configuration.
Attempting to sync time with chronyc.
Time synchronization was successful.
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/44]: creating directory server instance
  [2/44]: enabling ldapi
  [3/44]: configure autobind for root
...
...

So I'm not sure what really happened in Jan's test.

Comment 17 mreynolds 2018-07-23 15:06:18 UTC
(In reply to Alexander Bokovoy from comment #15)
> I haven't looked into details of the double logging, this is one of todo
> items for the PR:
> https://github.com/freeipa/freeipa/pull/1563#issuecomment-402403691

Well this is the fix that caused the regression in our CLI tools that I was just talking about.  So maybe it fixed an issue with FreeIPA, but it broke DS's CLI tool output.  At least it is unrelated to this issue...

Comment 18 Adam Williamson 2018-07-23 17:52:18 UTC
I figured out why this regression appears to have happened, if anyone's interested.

389-ds-base 1.4.0.11 package builds required 'perl(DSUtil)', which is provided only by 389-ds-base-legacy-tools. So effectively 389-ds-base required 389-ds-base-legacy-tools. 389-ds-base 1.4.0.12 package builds (for F28 and Rawhide) do *not* require 'perl(DSUtil)', so 389-ds-base does not require 389-ds-base-legacy-tools.

It seems freeipa-server never actually has had a dependency on 389-ds-base-legacy-tools, despite ipa-server-install using a command it contains. Probably the correct fix here would be for freeipa-server to grow that dependency (so long as it actually uses the tool, of course).

Comment 19 Lukas Slebodnik 2018-07-23 21:10:53 UTC
(In reply to Adam Williamson from comment #18)
> I figured out why this regression appears to have happened, if anyone's
> interested.
> 
> 389-ds-base 1.4.0.11 package builds required 'perl(DSUtil)', which is
> provided only by 389-ds-base-legacy-tools. So effectively 389-ds-base
> required 389-ds-base-legacy-tools. 389-ds-base 1.4.0.12 package builds (for
> F28 and Rawhide) do *not* require 'perl(DSUtil)', so 389-ds-base does not
> require 389-ds-base-legacy-tools.
> 
> It seems freeipa-server never actually has had a dependency on
> 389-ds-base-legacy-tools, despite ipa-server-install using a command it
> contains. Probably the correct fix here would be for freeipa-server to grow
> that dependency (so long as it actually uses the tool, of course).


As you can see in description even version in updates had this binary in *-legacy-tool

389-ds-base-legacy-tools-1.4.0.11-2.fc28.x86_64 : Legacy utilities for 389
                                                : Directory Server (%{variant})
Repo        : updates
Matched from:
Filename    : /usr/sbin/setup-ds.pl

And I could not see anything related in spec on f28
https://src.fedoraproject.org/rpms/389-ds-base/c/5eeb10f272238f500e9892e456dfb1872b53d802?branch=f28
https://src.fedoraproject.org/rpms/389-ds-base/c/49bf4301cae734ed24c187fd99df019404b7b3f8?branch=f28

But I agree that freeIPA server packages should require 389-ds-base-legacy-tools

BTW it works for me in f28 + updates-testing (I did not test rawhide)

Comment 20 Adam Williamson 2018-07-23 21:27:22 UTC
Lukas: "As you can see in description even version in updates had this binary in *-legacy-tool"

Yes - but until 1.4.0.11, 389-ds-base required 'perl(DSUtil)', which is only provided by 389-ds-base-legacy-tools. So any time you installed 389-ds-base, you got 389-ds-base-legacy-tools too. freeipa-server *should* have depended on 389-ds-base-legacy-tools at this time, but it got away with *not* depending on it, because when its dependency on 389-ds-base was fulfilled, 389-ds-base-legacy-tools happened to come along too, because of this dependency.

The reason things suddenly stopped working is that the 1.4.0.12 389-ds-base packages do *not* require perl(DSUtil). So 389-ds-base can now be installed *without* 389-ds-base-legacy-extras. So now when freeipa-server is installed, 389-ds-base is installed too, but 389-ds-base-legacy-extras is *not* any more.

Comment 21 Adam Williamson 2018-07-23 21:28:07 UTC
BTW, after working around this, openQA tests hit the segfault in ns-slapd too. I will try and get a traceback from it and file a new bug.

Comment 22 Adam Williamson 2018-07-23 21:34:00 UTC
The reason you don't see anything in the spec, btw, is that these are auto-generated dependencies. Fedora package build process has a script that auto-detects and adds perl dependencies to package specs, just like we have for C library dependencies. Obviously in 1.4.0.11 some perl script that 389-ds-base shipped used a perl module called DSUtil, while in 1.4.0.12, that script is gone or doesn't use that module any more.

BTW, arguably the 389-ds-base spec should be filtering out these automatic depends/provides anyway, as it actually installs these perl modules in a location that is not on the perl module path by default, so you can't just install 389-ds-base then write a perl script that does `use DSUtil;` - it won't be able to find the module. The script has to do `use lib qw(/usr/lib64/dirsrv/perl);` first, to add the directory to the module path. I'll file a separate bug on this, I guess...

Comment 23 Adam Williamson 2018-07-23 21:41:44 UTC
Filed https://bugzilla.redhat.com/show_bug.cgi?id=1607633 for filtering out these auto-generated requires/provides.

Comment 24 mreynolds 2018-07-23 21:46:24 UTC
Running setup-ds.pl alone works for me on rawhide, but running freeipa install does cause a crash triggered by running setup-ds.pl (rght after its imports a ldif file).  

Now I can reliably reproduce the crash by running ldif2db using ipa's boot.ldif.  I am investigating that crash right now...

Comment 25 Adam Williamson 2018-07-23 21:54:29 UTC
We really should file a new bug for that. I'll do it. I won't bother getting a backtrace since you have it reproduced already. Here: https://bugzilla.redhat.com/show_bug.cgi?id=1607635

Comment 26 mreynolds 2018-07-23 22:27:29 UTC
FYI the crash in ldif2db happens when the server is shutting down after the import when DS calls NSS_Shutdown(), then it crashes inside of NSS:

See more info in https://bugzilla.redhat.com/show_bug.cgi?id=1607635

Comment 27 Adam Williamson 2018-07-28 00:25:27 UTC
This is now fixed for Rawhide and ON_QA for F28, as the update has been edited with a new freeipa-server build that includes the dependency.

Comment 28 Fedora Update System 2018-07-28 00:27:18 UTC
389-ds-base-1.4.0.13-1.fc28 freeipa-4.7.0-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-f6a8f8036d

Comment 29 Fedora Update System 2018-08-19 02:25:56 UTC
389-ds-base-1.4.0.13-1.fc28, freeipa-4.7.0-1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.