Even if I explicitly add /usr/sbin/setup-ds.pl to the dnf operation, on rawhide things fail with

 [error] RuntimeError: failed to create DS instance CalledProcessError(Command ['/usr/sbin/setup-ds.pl', '--silent', '--logfile', '-', '-f', '/tmp/tmpls_vesa4'] returned non-zero exit status 1: '')
FreeIPA server configuration failed.

https://travis-ci.org/adelton/freeipa-container/jobs/407095858

Correct we now use "dscreate" instead of setup-ds.pl in 1.4.0. 

Or, you can install "389-ds-base-legacy-tools" package to get the old perl scripts including setup-ds.pl.  This change has been in the works for a long time - it has been part of the development plan for 1.4.0 from the very beginning.  IPA has been aware of this change, but you can still install the legacy tools package if you want to use the old perl installer.

Mark, look at the comment 1: installing 389-ds-base-legacy-tools aren't fixing the problem.

... and it would really have been good to coordinate with FreeIPA when the old method is not available for use by default. Pull request to switch to dscreate is still being reviewed, also pending changes in 389-ds side.

For the record, ipaserver-install.log has the following in it:

[23/Jul/2018:12:14:25.556762318 +0000] - INFO - ldbm_ancestorid_new_idl_create_index - import userRoot: Created ancestorid index (new idl).
[23/Jul/2018:12:14:25.565269653 +0000] - INFO - import_main_offline - import userRoot: Flushing caches...
[23/Jul/2018:12:14:25.573994410 +0000] - INFO - import_main_offline - import userRoot: Closing files...
[23/Jul/2018:12:14:25.645427122 +0000] - INFO - dblayer_pre_close - All database threads now stopped
[23/Jul/2018:12:14:25.647679208 +0000] - INFO - import_main_offline - import userRoot: Import complete.  Processed 1 entries in 1 seconds. (1.00 entries/sec)
/usr/sbin/ldif2db: line 116:   111 Segmentation fault      /usr/sbin/ns-slapd ldif2db -D /etc/dirsrv/slapd-EXAMPLE-TEST -n "userRoot" -i "/var/lib/dirsrv/boot.ldif"

[18/07/23:12:14:25] - [Setup] Fatal Error: Could not create directory server instance 'EXAMPLE-TEST'.
Error: Could not create directory server instance 'EXAMPLE-TEST'.
[18/07/23:12:14:25] - [Setup] Fatal Exiting . . .
Log file is '-'

Exiting . . .
Log file is '-'


2018-07-23T12:14:25Z DEBUG stderr=
2018-07-23T12:14:25Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/ipaserver/install/dsinstance.py", line 571, in __create_instance
    ipautil.run(args)
  File "/usr/lib/python3.7/site-packages/ipapython/ipautil.py", line 572, in run
    p.returncode, arg_string, output_log, error_log

Jan, this from running setup-ds.pl right?   Not sure why ldif2db would be crashing since this area of the code has not been touched, either way that would need to be fixed. 

Is there a core file to look at by any chance?

(In reply to Alexander Bokovoy from comment #4)
> ... and it would really have been good to coordinate with FreeIPA when the
> old method is not available for use by default. Pull request to switch to
> dscreate is still being reviewed, also pending changes in 389-ds side.

I'm sorry that this causing issues, but FreeIPA has known for a very long time that we were making this change.   There are also no outstanding issues with dscreate in DS - I think you are referring to the python openssl requirement.  That was also addressed several releases ago which was communicated with the FreeIPA team.

(In reply to mreynolds from comment #6)
> Jan, this from running setup-ds.pl right?

It's from running ipa-server-install. My understanding it is calls setup-ds.pl.

> Is there a core file to look at by any chance?

No. This is in container, cores are hard to get by.

Mark, the issue is that you are pushing a change to Fedora that is not coordinated on package level with freeipa, so you are guaranteed to break freeipa installation. For such cases Fedora has joint bodhi requests that can include multiple packages. This is what matters to Fedora, specifically.

FreeIPA 4.7 does not include the patches to support dscreate yet as they are not yet ready (review-wise). Thus, Fedora packages need to use ds-setup.pl and knowing about that, it is better to recall your bodhi update and fix dependencies.

It is all about coordination: removing dependencies without making sure they will be picked up by another package is a guaranteed way to kill deployments. Let's remove the dependency when it is known to be not needed.

(In reply to Alexander Bokovoy from comment #9)
> Mark, the issue is that you are pushing a change to Fedora that is not
> coordinated on package level with freeipa, so you are guaranteed to break
> freeipa installation. For such cases Fedora has joint bodhi requests that
> can include multiple packages. This is what matters to Fedora, specifically.
> 
> FreeIPA 4.7 does not include the patches to support dscreate yet as they are
> not yet ready (review-wise). Thus, Fedora packages need to use ds-setup.pl
> and knowing about that, it is better to recall your bodhi update and fix
> dependencies.
> 
> It is all about coordination: removing dependencies without making sure they
> will be picked up by another package is a guaranteed way to kill
> deployments. Let's remove the dependency when it is known to be not needed.

I said I was sorry, I was not being sarcastic.  This one slipped through the cracks - especially since we still provide setup-ds.pl in 389-ds-base-legacy-tools package.  That package could easily be made a requirement in the IPA spec file - is that an option or do you actually need me to revert the patch and rebuild DS?  Or, can I help review the dscreate PR in FreeIPA?  That PR should be a priority in FreeIPA anyway.

(In reply to Alexander Bokovoy from comment #3)
> Mark, look at the comment 1: installing 389-ds-base-legacy-tools aren't
> fixing the problem.

setup-ds.pl from the legacy tools package works for me.  If there a reproducible test case then please file a new bug.

(In reply to mreynolds from comment #10)
> I said I was sorry, I was not being sarcastic.  This one slipped through the
> cracks - especially since we still provide setup-ds.pl in
> 389-ds-base-legacy-tools package.  That package could easily be made a
> requirement in the IPA spec file - is that an option or do you actually need
> me to revert the patch and rebuild DS?  Or, can I help review the dscreate
> PR in FreeIPA?  That PR should be a priority in FreeIPA anyway.
Thanks.

Thomas W. is working right now on FreeIPA 4.7.0 packages for rawhide and F28, they will include the Requires: 389-ds-base-legacy-tools there. Hopefully, this would happen today/tomorrow. I'd appreciate if we could add freeipa package to the same updates bodhi request in F28 so that people don't get affected by the change (aside from updates-testing right now).

As for the PR, there few changes that need to be done there to complete review as 389-ds now includes all required fixes to avoid the double logging. Once these are done, we can push the PR upstream. However, it is not going to be part of 4.7.0 as the release has already been done.

(In reply to mreynolds from comment #11)
> (In reply to Alexander Bokovoy from comment #3)
> > Mark, look at the comment 1: installing 389-ds-base-legacy-tools aren't
> > fixing the problem.
> 
> setup-ds.pl from the legacy tools package works for me.  If there a
> reproducible test case then please file a new bug.

FYI, the crash that was reported in comment 1 is not related to setup-ds.pl, but actually the ldif2db utility.  So it is unrelated to this change, and will probably still be a problem after I revert the legacy tool package.  It needs more investigation...

(In reply to Alexander Bokovoy from comment #12)
> (In reply to mreynolds from comment #10)
> > I said I was sorry, I was not being sarcastic.  This one slipped through the
> > cracks - especially since we still provide setup-ds.pl in
> > 389-ds-base-legacy-tools package.  That package could easily be made a
> > requirement in the IPA spec file - is that an option or do you actually need
> > me to revert the patch and rebuild DS?  Or, can I help review the dscreate
> > PR in FreeIPA?  That PR should be a priority in FreeIPA anyway.
> Thanks.
> 
> Thomas W. is working right now on FreeIPA 4.7.0 packages for rawhide and
> F28, they will include the Requires: 389-ds-base-legacy-tools there.
> Hopefully, this would happen today/tomorrow. I'd appreciate if we could add
> freeipa package to the same updates bodhi request in F28 so that people
> don't get affected by the change (aside from updates-testing right now).
> 
> As for the PR, there few changes that need to be done there to complete
> review as 389-ds now includes all required fixes to avoid the double
> logging. 

Double logging, is this a new bug or a old one?  We are seeing "double logging" in lib389 cli tools after we very recently got a contribution from a community member (not fixed yet).  Just checking if it's the same thing, or we still have a blocker for ipa.

I haven't looked into details of the double logging, this is one of todo items for the PR: https://github.com/freeipa/freeipa/pull/1563#issuecomment-402403691

Also, I just installed freeipa 4.6.90 on F28 with legacy tools and it succeeded:

Synchronizing time
No SRV records of NTP servers found and no NTP server or pool address was provided.
Using default chrony configuration.
Attempting to sync time with chronyc.
Time synchronization was successful.
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/44]: creating directory server instance
  [2/44]: enabling ldapi
  [3/44]: configure autobind for root
...
...

So I'm not sure what really happened in Jan's test.

(In reply to Alexander Bokovoy from comment #15)
> I haven't looked into details of the double logging, this is one of todo
> items for the PR:
> https://github.com/freeipa/freeipa/pull/1563#issuecomment-402403691

Well this is the fix that caused the regression in our CLI tools that I was just talking about.  So maybe it fixed an issue with FreeIPA, but it broke DS's CLI tool output.  At least it is unrelated to this issue...

I figured out why this regression appears to have happened, if anyone's interested.

389-ds-base 1.4.0.11 package builds required 'perl(DSUtil)', which is provided only by 389-ds-base-legacy-tools. So effectively 389-ds-base required 389-ds-base-legacy-tools. 389-ds-base 1.4.0.12 package builds (for F28 and Rawhide) do *not* require 'perl(DSUtil)', so 389-ds-base does not require 389-ds-base-legacy-tools.

It seems freeipa-server never actually has had a dependency on 389-ds-base-legacy-tools, despite ipa-server-install using a command it contains. Probably the correct fix here would be for freeipa-server to grow that dependency (so long as it actually uses the tool, of course).

(In reply to Adam Williamson from comment #18)
> I figured out why this regression appears to have happened, if anyone's
> interested.
> 
> 389-ds-base 1.4.0.11 package builds required 'perl(DSUtil)', which is
> provided only by 389-ds-base-legacy-tools. So effectively 389-ds-base
> required 389-ds-base-legacy-tools. 389-ds-base 1.4.0.12 package builds (for
> F28 and Rawhide) do *not* require 'perl(DSUtil)', so 389-ds-base does not
> require 389-ds-base-legacy-tools.
> 
> It seems freeipa-server never actually has had a dependency on
> 389-ds-base-legacy-tools, despite ipa-server-install using a command it
> contains. Probably the correct fix here would be for freeipa-server to grow
> that dependency (so long as it actually uses the tool, of course).


As you can see in description even version in updates had this binary in *-legacy-tool

389-ds-base-legacy-tools-1.4.0.11-2.fc28.x86_64 : Legacy utilities for 389
                                                : Directory Server (%{variant})
Repo        : updates
Matched from:
Filename    : /usr/sbin/setup-ds.pl

And I could not see anything related in spec on f28
https://src.fedoraproject.org/rpms/389-ds-base/c/5eeb10f272238f500e9892e456dfb1872b53d802?branch=f28
https://src.fedoraproject.org/rpms/389-ds-base/c/49bf4301cae734ed24c187fd99df019404b7b3f8?branch=f28

But I agree that freeIPA server packages should require 389-ds-base-legacy-tools

BTW it works for me in f28 + updates-testing (I did not test rawhide)

Lukas: "As you can see in description even version in updates had this binary in *-legacy-tool"

Yes - but until 1.4.0.11, 389-ds-base required 'perl(DSUtil)', which is only provided by 389-ds-base-legacy-tools. So any time you installed 389-ds-base, you got 389-ds-base-legacy-tools too. freeipa-server *should* have depended on 389-ds-base-legacy-tools at this time, but it got away with *not* depending on it, because when its dependency on 389-ds-base was fulfilled, 389-ds-base-legacy-tools happened to come along too, because of this dependency.

The reason things suddenly stopped working is that the 1.4.0.12 389-ds-base packages do *not* require perl(DSUtil). So 389-ds-base can now be installed *without* 389-ds-base-legacy-extras. So now when freeipa-server is installed, 389-ds-base is installed too, but 389-ds-base-legacy-extras is *not* any more.

BTW, after working around this, openQA tests hit the segfault in ns-slapd too. I will try and get a traceback from it and file a new bug.

The reason you don't see anything in the spec, btw, is that these are auto-generated dependencies. Fedora package build process has a script that auto-detects and adds perl dependencies to package specs, just like we have for C library dependencies. Obviously in 1.4.0.11 some perl script that 389-ds-base shipped used a perl module called DSUtil, while in 1.4.0.12, that script is gone or doesn't use that module any more.

BTW, arguably the 389-ds-base spec should be filtering out these automatic depends/provides anyway, as it actually installs these perl modules in a location that is not on the perl module path by default, so you can't just install 389-ds-base then write a perl script that does `use DSUtil;` - it won't be able to find the module. The script has to do `use lib qw(/usr/lib64/dirsrv/perl);` first, to add the directory to the module path. I'll file a separate bug on this, I guess...

Filed https://bugzilla.redhat.com/show_bug.cgi?id=1607633 for filtering out these auto-generated requires/provides.

Running setup-ds.pl alone works for me on rawhide, but running freeipa install does cause a crash triggered by running setup-ds.pl (rght after its imports a ldif file).  

Now I can reliably reproduce the crash by running ldif2db using ipa's boot.ldif.  I am investigating that crash right now...

We really should file a new bug for that. I'll do it. I won't bother getting a backtrace since you have it reproduced already. Here: https://bugzilla.redhat.com/show_bug.cgi?id=1607635

FYI the crash in ldif2db happens when the server is shutting down after the import when DS calls NSS_Shutdown(), then it crashes inside of NSS:

See more info in https://bugzilla.redhat.com/show_bug.cgi?id=1607635

This is now fixed for Rawhide and ON_QA for F28, as the update has been edited with a new freeipa-server build that includes the dependency.

389-ds-base-1.4.0.13-1.fc28 freeipa-4.7.0-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-f6a8f8036d

389-ds-base-1.4.0.13-1.fc28, freeipa-4.7.0-1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.