Bug 971509 - Failed to initialize the realm container
Failed to initialize the realm container
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: libbsd (Show other bugs)
19
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Eric Smith
Fedora Extras Quality Assurance
:
Depends On: 971513
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-06 12:41 EDT by Dean Hunter
Modified: 2013-06-13 02:44 EDT (History)
5 users (show)

See Also:
Fixed In Version: libbsd-0.5.2-1.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-13 02:44:46 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
ipaserver-install.log (309.97 KB, text/plain)
2013-06-06 12:42 EDT, Dean Hunter
no flags Details
ipaserver-install.log showing failure at step 6 of KDC setup (300.55 KB, text/plain)
2013-06-06 13:20 EDT, Eric Smith
no flags Details

  None (edit)
Description Dean Hunter 2013-06-06 12:41:45 EDT
Description of problem:

While rebuilding FreeIPA server using Fedora 19 beta ipa-server-install fails.


Version-Release number of selected component (if applicable):

================================================================================
 Package                   Arch   Version                 Repository       Size
================================================================================
Installing:
 bind                      x86_64 32:9.9.3-3.P1.fc19      updates-testing 1.7 M
 bind-dyndb-ldap           x86_64 3.3-1.fc19              updates-testing  90 k
 freeipa-server            x86_64 3.2.0-2.fc19            fedora          1.1 M


How reproducible:


Steps to Reproduce:

1. Build a new Fedora 19 Beta VM
2. yum install --assumeyes bind bind-dyndb-ldap freeipa-server
3. ipa-server-install \
     --admin-password adminpassword \
     --domain hunter.org \
     --ds-password dspassword \
     --hostname ipa.hunter.org \
     --idstart 128000000 \
     --mkhomedir \
     --no-forwarders \
     --no-ntp \
     --realm HUNTER.ORG \
     --setup-dns \
     --ssh-trust-dns \
     --unattended


Actual results:

...
Configuring Kerberos KDC (krb5kdc): Estimated time 30 seconds
  [1/10]: adding sasl mappings to the directory
  [2/10]: adding kerberos container to the directory
  [3/10]: configuring KDC
  [4/10]: initialize kerberos container
Failed to initialize the realm container
  [5/10]: adding default ACIs
  [6/10]: creating a keytab for the directory
  [7/10]: creating a keytab for the machine
  [8/10]: adding the password extension to the directory
  [9/10]: starting the KDC
  [10/10]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc)Unexpected error - see /var/log/ipaserver-install.log for details:
CalledProcessError: Command '/bin/systemctl restart kadmin.service' returned non-zero exit status 1
.
Configuring kadmin
....


Expected results: No errors


Additional info:
Comment 1 Dean Hunter 2013-06-06 12:42:31 EDT
Created attachment 757755 [details]
ipaserver-install.log
Comment 2 Alexander Bokovoy 2013-06-06 12:48:27 EDT
There is regression in libbsd that caused this issue. Please downgrade and see https://admin.fedoraproject.org/updates/FEDORA-2013-10062/libbsd-0.5.1-1.fc19
Comment 3 Eric Smith 2013-06-06 13:18:28 EDT
I'll try to find the problem in libbsd. In the mean time I've unpushed the update.

Thanks for providing the detailed instructions to reproduce the problem. I've tried it, and I get a failure before the one you describe, at step 6 of "Configuring Kerberos KDC":

Configuring Kerberos KDC (krb5kdc): Estimated time 30 seconds
  [1/10]: adding sasl mappings to the directory
  [2/10]: adding kerberos container to the directory
  [3/10]: configuring KDC
  [4/10]: initialize kerberos container
Failed to initialize the realm container
  [5/10]: adding default ACIs
  [6/10]: creating a keytab for the directory
Unexpected error - see /var/log/ipaserver-install.log for details:
CalledProcessError: Command 'kadmin.local -q addprinc -randkey ldap/ipa.z00.us@Z00.US -x ipa-setup-override-restrictions' returned non-zero exit status 1

This is probably some sort of operator error on my part, but I'm not sure how to resolve it, so any advice will be appreciated.
Comment 4 Eric Smith 2013-06-06 13:20:17 EDT
Created attachment 757779 [details]
ipaserver-install.log showing failure at step 6 of KDC setup

I think this is probably not actually related to this bug, but I need to know how to get past this in order to reproduce the actual problem in libbsd-0.5.1.
Comment 5 Rob Crittenden 2013-06-06 13:27:11 EDT
The failure occurs when krb5_util is run. It segfaults when creating the Kerberos database which then causes kadmin.local to fail later. We should probably be catching this failure in the IPA installer so it fails at the right place (the "Failed to initialize realm container" message above.

The only way we've been able to get past this failure so far is to downgrade libbsd to 0.4.
Comment 6 Eric Smith 2013-06-06 13:33:25 EDT
I understand that, but following the given instructions doesn't even get me that far.  I get a failure at step 6 of KDC setup, not step 10.  That's why I was requesting info on how to get past that.
Comment 7 Dean Hunter 2013-06-06 13:37:42 EDT
The error does not occur in step 10. As Rob noted there is a bug in ipa-server-install such that it does not stop when the failure occurs in step 4 or perhaps even earlier.
Comment 8 Eric Smith 2013-06-06 13:46:26 EDT
I don't understand why the output you included from the command shows steps up through 10, while the output I get stops at 6 with a different error reported.
Comment 9 Dean Hunter 2013-06-06 13:52:28 EDT
I do not understand either. But the error I am reporting is in step 4.
Comment 10 Alexander Bokovoy 2013-06-06 15:18:37 EDT
Eric, I think we found the issue.

IPA KDB driver is linked against Samba libraries. These libraries, in turn, are linked against Samba's internal library that provides replacement for functions known to be broken or not implemented on the platform. Some code in that library (libreplace) uses libbsd.

libbsd 0.5.1 has setproctitle() replacement. The code in src/setproctitle.c sets up spt_init() as a constructor that is called on library initialization.

spt_init() calls internally environ handling code, to copy it. spt_copyenv() calls to spt_clearenv(). spt_clearenv() calls system-provided clearenv() if that is there.

Here is where the problem comes. Glibc's clearenv() frees environment memory if that was allocated by the glibc (as in our case). It also always returns 0. Any allocation for the new environment will happen at first setenv() call.

At this point spt_copyenv() attempts to access memory which was freed by Glibc's clearenv() and may or may not crash. For Nalin under valgrind it crashed, so we've got following backtrace (I'm cutting down to interesting parts):

#0  __strchr_sse2 () at ../sysdeps/x86_64/strchr.S:32
#1  0x00007fffec389cdb in spt_copyenv (oldenv=0x55555577ec10)
    at setproctitle.c:94
#2  spt_init (argc=8, argv=0x7fffffffe448, envp=0x55555577ec10)
    at setproctitle.c:172
#3  0x00007ffff7deb4f3 in call_init (env=0x55555577ec10, argv=0x7fffffffe448, 
    argc=8, l=<optimized out>) at dl-init.c:82
#4  _dl_init (main_map=main_map@entry=0x5555557baac0, argc=8, 
    argv=0x7fffffffe448, env=0x55555577ec10) at dl-init.c:131
#5  0x00007ffff7defb04 in dl_open_worker (a=a@entry=0x7fffffffda98)
    at dl-open.c:566
#6  0x00007ffff7deb304 in _dl_catch_error (
    objname=objname@entry=0x7fffffffda88, 
    errstring=errstring@entry=0x7fffffffda90, 
    mallocedp=mallocedp@entry=0x7fffffffda80, 
    operate=operate@entry=0x7ffff7def770 <dl_open_worker>, 
    args=args@entry=0x7fffffffda98) at dl-error.c:177
#7  0x00007ffff7def21b in _dl_open (
    file=0x5555557baa40 "/usr/lib64/krb5/plugins/kdb/ipadb.so", 
    mode=-2147483646, caller_dlopen=0x7ffff6a143cd <krb5int_open_plugin+349>, 
    nsid=-2, argc=8, argv=<optimized out>, env=0x55555577ec10) at dl-open.c:656
#8  0x00007ffff733502b in dlopen_doit (a=a@entry=0x7fffffffdca0) at dlopen.c:66


So, spt_clearenv() needs to be smarter when it uses system-provided clearenv() and spt_copyenv() should be careful -- oldenv will likely be deleted before first use.
Comment 11 Alexander Bokovoy 2013-06-06 15:20:11 EDT
Moving to libbsd.
Comment 12 Eric Smith 2013-06-06 15:58:58 EDT
I reported the bug upstream.  See bug #971513 for further info.
Comment 13 Fedora Update System 2013-06-06 16:47:18 EDT
libbsd-0.5.1-2.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/libbsd-0.5.1-2.fc19
Comment 14 Dean Hunter 2013-06-06 18:14:21 EDT
libbsd-0.5.1-2.fc19 resolves the problem for me:

  yum install --assumeyes createrepo koji

  mkdir /root/local
  cd /root/local
  koji download-build --arch x86_64 libbsd-0.5.1-2.fc19
  cd -
  createrepo /root/local

  cat >/etc/yum.repos.d/local.repo <<EOD
[local]
baseurl=file:///root/local
enabled=0
gpgcheck=0
name=Local Updates $releasever - $basearch
EOD

  yum install --assumeyes --enablerepo local bind bind-dyndb-ldap freeipa-server
...
Updating for dependencies:
 bind-libs                 x86_64 32:9.9.3-3.P1.fc19      updates-testing 978 k
 bind-libs-lite            x86_64 32:9.9.3-3.P1.fc19      updates-testing 693 k
 bind-license              noarch 32:9.9.3-3.P1.fc19      updates-testing  77 k
 libbsd                    x86_64 0.5.1-2.fc19            local            61 k
 libldb                    x86_64 1.1.15-3.fc19           updates-testing 119 k
 libtdb                    x86_64 1.2.12-1.fc19           updates-testing  38 k

Transaction Summary
....
Comment 15 Dean Hunter 2013-06-06 18:16:14 EDT
Just for my education, is there a way to tell koji where to download files instead of the current directory?
Comment 16 Fedora Update System 2013-06-07 11:37:38 EDT
Package libbsd-0.5.1-2.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libbsd-0.5.1-2.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-10259/libbsd-0.5.1-2.fc19
then log in and leave karma (feedback).
Comment 17 Fedora Update System 2013-06-10 15:48:29 EDT
libbsd-0.5.2-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/libbsd-0.5.2-1.fc19
Comment 18 Fedora Update System 2013-06-13 02:44:46 EDT
libbsd-0.5.2-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.