Bug 886162 - Cannot connect to LDAP to add DNS records during yum update of freeipa-server
Summary: Cannot connect to LDAP to add DNS records during yum update of freeipa-server
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Fedora
Classification: Fedora
Component: freeipa
Version: 18
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Rob Crittenden
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-12-11 16:26 UTC by Dean Hunter
Modified: 2013-01-24 22:08 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-01-24 22:08:31 UTC
Type: Bug


Attachments (Terms of Use)

Description Dean Hunter 2012-12-11 16:26:36 UTC
Description of problem:
While applying the latest updates (yum update) an error occured during the update of Freeipa-server.


Version-Release number of selected component (if applicable):
freeipa-server-3.1.0-1.fc18.x86_64


How reproducible:
Unknown


Steps to Reproduce:
1. yum update
2.
3.
  
Actual results:
  Updating   : pki-ca-10.0.0-2.fc18.noarch                                17/72 
  Updating   : freeipa-server-3.1.0-1.fc18.x86_64                         18/72 
Found IPA server for domain HUNTER.ORG
Converting services setup to systemd
    Upgrade /etc/sysconfig/dirsrv
    Upgrade /etc/sysconfig/krb5kdc
    Re-enable Directory server instances PKI-IPA and HUNTER-ORG 
    Re-enable IPA service
Finished.
Cannot connect to LDAP to add DNS records: cannot connect to u'ldapi://%2fvar%2frun%2fslapd-HUNTER-ORG.socket': LDAP Server Down
Unexpected error
CalledProcessError: Command '/bin/systemctl restart named.service' returned non-zero exit status 1
  Updating   : freeipa-server-selinux-3.1.0-1.fc18.x86_64                 19/72 
  Updating   : policycoreutils-python-2.1.13-44.fc18.x86_64               20/72 


Expected results:
No errors

Additional info:

Comment 1 Dean Hunter 2012-12-11 16:30:56 UTC
From /var/log/messages:

Dec 11 10:06:10 server yum[14034]: Updated: pki-server-10.0.0-2.fc18.noarch
Dec 11 10:06:11 server systemd[1]: Reloading.
Dec 11 10:06:11 server yum[14034]: Updated: pki-ca-10.0.0-2.fc18.noarch
Dec 11 10:06:13 server systemd[1]: Reloading.
Dec 11 10:06:14 server systemd[1]: Reloading.
Dec 11 10:06:14 server systemd[1]: Reloading.
Dec 11 10:06:14 server systemd[1]: Reloading.
Dec 11 10:06:14 server systemd[1]: Reloading.
Dec 11 10:06:14 server systemd[1]: Reloading.
Dec 11 10:06:14 server systemd[1]: Stopping Identity, Policy, Audit...
Dec 11 10:06:15 server systemd[1]: Stopping PKI Tomcat Server pki-tomcat...
Dec 11 10:06:15 server pkidaemon[14291]: An exit status of '143' refers to the 'systemd' method of using 'SIGTERM' to shutdown a Java process and can safely be ignored.
Dec 11 10:06:16 server systemd[1]: pki-tomcatd@pki-tomcat.service: main process exited, code=exited, status=143/n/a
Dec 11 10:06:16 server systemd[1]: Stopped PKI Tomcat Server pki-tomcat.
Dec 11 10:06:16 server systemd[1]: Unit pki-tomcatd@pki-tomcat.service entered failed state
Dec 11 10:06:16 server systemd[1]: Stopping PKI Tomcat Server.
Dec 11 10:06:16 server systemd[1]: Stopped target PKI Tomcat Server.
Dec 11 10:06:16 server systemd[1]: Stopping The Apache HTTP Server...
Dec 11 10:06:16 server systemd[1]: Stopped The Apache HTTP Server.
Dec 11 10:06:16 server systemd[1]: Stopping IPA memcached daemon, increases IPA server performance...
Dec 11 10:06:16 server systemd[1]: Stopped IPA memcached daemon, increases IPA server performance.
Dec 11 10:06:16 server systemd[1]: Stopping Berkeley Internet Name Domain (DNS)...
Dec 11 10:06:16 server named[1645]: shutting down
Dec 11 10:06:16 server named[1645]: no longer listening on ::#53
Dec 11 10:06:16 server named[1645]: no longer listening on 127.0.0.1#53
Dec 11 10:06:16 server named[1645]: no longer listening on 192.168.1.11#53
Dec 11 10:06:41 server systemd[1]: named.service stopping timed out. Killing.
Dec 11 10:06:41 server systemd[1]: named.service: main process exited, code=killed, status=9/KILL
Dec 11 10:06:41 server systemd[1]: Stopped Berkeley Internet Name Domain (DNS).
Dec 11 10:06:41 server systemd[1]: Unit named.service entered failed state
Dec 11 10:06:41 server systemd[1]: Stopping Kerberos 5 Password-changing and Administration...
Dec 11 10:06:41 server systemd[1]: kadmin.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 11 10:06:41 server systemd[1]: Stopped Kerberos 5 Password-changing and Administration.
Dec 11 10:06:41 server systemd[1]: Unit kadmin.service entered failed state
Dec 11 10:06:41 server systemd[1]: Stopping Kerberos 5 KDC...
Dec 11 10:06:41 server systemd[1]: Stopped Kerberos 5 KDC.
Dec 11 10:06:41 server systemd[1]: Stopping 389 Directory Server PKI-IPA....
Dec 11 10:06:41 server systemd[1]: Stopping 389 Directory Server HUNTER-ORG....
Dec 11 10:06:43 server systemd[1]: Stopped 389 Directory Server HUNTER-ORG..
Dec 11 10:06:43 server systemd[1]: Stopped 389 Directory Server PKI-IPA..
Dec 11 10:06:43 server systemd[1]: Stopping 389 Directory Server.
Dec 11 10:06:43 server systemd[1]: Stopped target 389 Directory Server.
Dec 11 10:06:43 server ipactl[14284]: ipa: INFO: The ipactl command was successful
Dec 11 10:06:43 server ipactl[14284]: Stopping pki-tomcatd Service
Dec 11 10:06:43 server ipactl[14284]: Stopping httpd Service
Dec 11 10:06:43 server ipactl[14284]: Stopping ipa_memcached Service
Dec 11 10:06:43 server ipactl[14284]: Stopping named Service
Dec 11 10:06:43 server ipactl[14284]: Stopping kadmin Service
Dec 11 10:06:43 server ipactl[14284]: Stopping krb5kdc Service
Dec 11 10:06:43 server ipactl[14284]: Stopping Directory Service
Dec 11 10:06:44 server systemd[1]: Starting Identity, Policy, Audit...
Dec 11 10:06:45 server systemd[1]: Starting 389 Directory Server.
Dec 11 10:06:45 server systemd[1]: Reached target 389 Directory Server.
Dec 11 10:06:45 server systemd[1]: Starting 389 Directory Server PKI-IPA....
Dec 11 10:06:45 server systemd[1]: Starting 389 Directory Server HUNTER-ORG....
Dec 11 10:06:45 server ns-slapd: auxpropfunc error version mismatch with plug-in
Dec 11 10:06:45 server ns-slapd: auxpropfunc error version mismatch with plug-in
Dec 11 10:06:45 server systemd[1]: Started 389 Directory Server PKI-IPA..
Dec 11 10:06:45 server systemd[1]: Started 389 Directory Server HUNTER-ORG..
Dec 11 10:06:46 server systemd[1]: Starting Kerberos 5 KDC...
Dec 11 10:06:46 server systemd[1]: Started Kerberos 5 KDC.
Dec 11 10:06:46 server systemd[1]: Starting Kerberos 5 Password-changing and Administration...
Dec 11 10:06:46 server systemd[1]: Started Kerberos 5 Password-changing and Administration.
Dec 11 10:06:46 server systemd[1]: Starting Berkeley Internet Name Domain (DNS)...
Dec 11 10:06:46 server named-checkconf[14504]: zone localhost.localdomain/IN: loaded serial 0
Dec 11 10:06:46 server named-checkconf[14504]: zone localhost/IN: loaded serial 0
Dec 11 10:06:46 server named-checkconf[14504]: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0
Dec 11 10:06:46 server named-checkconf[14504]: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0
Dec 11 10:06:46 server named-checkconf[14504]: zone 0.in-addr.arpa/IN: loaded serial 0
Dec 11 10:06:46 server named[14510]: starting BIND 9.9.2-P1-RedHat-9.9.2-5.P1.fc18 -u named
Dec 11 10:06:46 server named[14510]: built with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-libtool' '--localstatedir=/var' '--enable-threads' '--enable-ipv6' '--with-pic' '--disable-static' '--disable-openssl-version-check' '--enable-exportlib' '--with-export-libdir=/usr/lib64' '--with-export-includedir=/usr/include' '--includedir=/usr/include/bind9' '--with-pkcs11=/usr/lib64/pkcs11/PKCS11_API.so' '--with-dlz-ldap=yes' '--with-dlz-postgres=yes' '--with-dlz-mysql=yes' '--with-dlz-filesystem=yes' '--with-dlz-bdb=yes' '--with-gssapi=yes' '--disable-isc-spnego' '--enable-fixed-rrset' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro ' 'CPPFLAGS= -DDIG_SIGCHASE'
Dec 11 10:06:46 server named[14510]: ----------------------------------------------------
Dec 11 10:06:46 server named[14510]: BIND 9 is maintained by Internet Systems Consortium,
Dec 11 10:06:46 server named[14510]: Inc. (ISC), a non-profit 501(c)(3) public-benefit
Dec 11 10:06:46 server named[14510]: corporation.  Support and training for BIND 9 are
Dec 11 10:06:46 server named[14510]: available at https://www.isc.org/support
Dec 11 10:06:46 server named[14510]: ----------------------------------------------------
Dec 11 10:06:46 server named[14510]: adjusted limit on open files from 4096 to 1048576
Dec 11 10:06:46 server named[14510]: found 8 CPUs, using 8 worker threads
Dec 11 10:06:46 server named[14510]: using 8 UDP listeners per interface
Dec 11 10:06:46 server named[14510]: using up to 4096 sockets
Dec 11 10:06:46 server named[14510]: loading configuration from '/etc/named.conf'
Dec 11 10:06:46 server named[14510]: using default UDP/IPv4 port range: [1024, 65535]
Dec 11 10:06:46 server named[14510]: using default UDP/IPv6 port range: [1024, 65535]
Dec 11 10:06:46 server named[14510]: listening on IPv6 interfaces, port 53
Dec 11 10:06:46 server named[14510]: listening on IPv4 interface lo, 127.0.0.1#53
Dec 11 10:06:46 server named[14510]: listening on IPv4 interface em1, 192.168.1.11#53
Dec 11 10:06:46 server named[14510]: generating session key for dynamic DNS
Dec 11 10:06:46 server named[14510]: sizing zone task pool based on 6 zones
Dec 11 10:06:46 server named[14510]: /etc/named.conf:12: no forwarders seen; disabling forwarding
Dec 11 10:06:46 server named[14510]: set up managed keys zone for view _default, file 'managed-keys.bind'
Dec 11 10:06:57 server named[14510]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Cannot resolve servers for KDC in realm "HUNTER.ORG")
Dec 11 10:06:57 server named[14510]: bind to LDAP server failed: Local error
Dec 11 10:06:57 server named[14510]: loading configuration: failure
Dec 11 10:06:57 server named[14510]: exiting (due to fatal error)
Dec 11 10:06:57 server systemd[1]: named.service: control process exited, code=exited status=1
Dec 11 10:06:57 server systemd[1]: Failed to start Berkeley Internet Name Domain (DNS).
Dec 11 10:06:57 server systemd[1]: Unit named.service entered failed state
Dec 11 10:06:57 server ipactl[14395]: Job for named.service failed. See 'systemctl status named.service' and 'journalctl -n' for details.
Dec 11 10:06:57 server ipactl[14395]: Failed to start named Service
Dec 11 10:06:57 server ipactl[14395]: Shutting down
Dec 11 10:06:57 server systemd[1]: Stopping Kerberos 5 KDC...
Dec 11 10:06:57 server systemd[1]: Stopped Kerberos 5 KDC.
Dec 11 10:06:57 server systemd[1]: Stopping Kerberos 5 Password-changing and Administration...
Dec 11 10:06:57 server systemd[1]: kadmin.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 11 10:06:57 server systemd[1]: Stopped Kerberos 5 Password-changing and Administration.
Dec 11 10:06:57 server systemd[1]: Unit kadmin.service entered failed state
Dec 11 10:06:57 server systemd[1]: Stopped Berkeley Internet Name Domain (DNS).
Dec 11 10:06:57 server systemd[1]: Stopped IPA memcached daemon, increases IPA server performance.
Dec 11 10:06:57 server systemd[1]: Stopped The Apache HTTP Server.
Dec 11 10:06:57 server systemd[1]: Stopped target PKI Tomcat Server.
Dec 11 10:06:57 server systemd[1]: Stopping 389 Directory Server PKI-IPA....
Dec 11 10:06:57 server systemd[1]: Stopping 389 Directory Server HUNTER-ORG....
Dec 11 10:06:58 server systemd[1]: Stopped 389 Directory Server PKI-IPA..
Dec 11 10:06:58 server systemd[1]: Stopped 389 Directory Server HUNTER-ORG..
Dec 11 10:06:58 server systemd[1]: Stopping 389 Directory Server.
Dec 11 10:06:58 server systemd[1]: Stopped target 389 Directory Server.
Dec 11 10:06:58 server ipactl[14395]: ipa: INFO:   File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 617, in run_script
Dec 11 10:06:58 server ipactl[14395]: return_value = main_function()
Dec 11 10:06:58 server ipactl[14395]: File "/usr/sbin/ipactl", line 490, in main
Dec 11 10:06:58 server ipactl[14395]: ipa_start(options)
Dec 11 10:06:58 server ipactl[14395]: File "/usr/sbin/ipactl", line 261, in ipa_start
Dec 11 10:06:58 server ipactl[14395]: raise IpactlError("Aborting ipactl")
Dec 11 10:06:58 server ipactl[14395]: ipa: INFO: The ipactl command failed, exception: IpactlError: Aborting ipactl
Dec 11 10:06:58 server ipactl[14395]: Aborting ipactl
Dec 11 10:06:58 server ipactl[14395]: Starting Directory Service
Dec 11 10:06:58 server ipactl[14395]: Starting krb5kdc Service
Dec 11 10:06:58 server ipactl[14395]: Starting kadmin Service
Dec 11 10:06:58 server ipactl[14395]: Starting named Service
Dec 11 10:06:58 server systemd[1]: ipa.service: main process exited, code=exited, status=1/FAILURE
Dec 11 10:06:58 server systemd[1]: Failed to start Identity, Policy, Audit.
Dec 11 10:06:58 server systemd[1]: Unit ipa.service entered failed state
Dec 11 10:07:00 server systemd[1]: Starting Berkeley Internet Name Domain (DNS)...
Dec 11 10:07:00 server named-checkconf[14554]: zone localhost.localdomain/IN: loaded serial 0
Dec 11 10:07:00 server named-checkconf[14554]: zone localhost/IN: loaded serial 0
Dec 11 10:07:00 server named-checkconf[14554]: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0
Dec 11 10:07:00 server named-checkconf[14554]: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0
Dec 11 10:07:00 server named-checkconf[14554]: zone 0.in-addr.arpa/IN: loaded serial 0
Dec 11 10:07:00 server named[14558]: starting BIND 9.9.2-P1-RedHat-9.9.2-5.P1.fc18 -u named
Dec 11 10:07:00 server named[14558]: built with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-libtool' '--localstatedir=/var' '--enable-threads' '--enable-ipv6' '--with-pic' '--disable-static' '--disable-openssl-version-check' '--enable-exportlib' '--with-export-libdir=/usr/lib64' '--with-export-includedir=/usr/include' '--includedir=/usr/include/bind9' '--with-pkcs11=/usr/lib64/pkcs11/PKCS11_API.so' '--with-dlz-ldap=yes' '--with-dlz-postgres=yes' '--with-dlz-mysql=yes' '--with-dlz-filesystem=yes' '--with-dlz-bdb=yes' '--with-gssapi=yes' '--disable-isc-spnego' '--enable-fixed-rrset' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro ' 'CPPFLAGS= -DDIG_SIGCHASE'
Dec 11 10:07:00 server named[14558]: ----------------------------------------------------
Dec 11 10:07:00 server named[14558]: BIND 9 is maintained by Internet Systems Consortium,
Dec 11 10:07:00 server named[14558]: Inc. (ISC), a non-profit 501(c)(3) public-benefit
Dec 11 10:07:00 server named[14558]: corporation.  Support and training for BIND 9 are
Dec 11 10:07:00 server named[14558]: available at https://www.isc.org/support
Dec 11 10:07:00 server named[14558]: ----------------------------------------------------
Dec 11 10:07:00 server named[14558]: adjusted limit on open files from 4096 to 1048576
Dec 11 10:07:00 server named[14558]: found 8 CPUs, using 8 worker threads
Dec 11 10:07:00 server named[14558]: using 8 UDP listeners per interface
Dec 11 10:07:00 server named[14558]: using up to 4096 sockets
Dec 11 10:07:00 server named[14558]: loading configuration from '/etc/named.conf'
Dec 11 10:07:00 server named[14558]: using default UDP/IPv4 port range: [1024, 65535]
Dec 11 10:07:00 server named[14558]: using default UDP/IPv6 port range: [1024, 65535]
Dec 11 10:07:00 server named[14558]: listening on IPv6 interfaces, port 53
Dec 11 10:07:00 server named[14558]: listening on IPv4 interface lo, 127.0.0.1#53
Dec 11 10:07:00 server named[14558]: listening on IPv4 interface em1, 192.168.1.11#53
Dec 11 10:07:00 server named[14558]: generating session key for dynamic DNS
Dec 11 10:07:00 server named[14558]: sizing zone task pool based on 6 zones
Dec 11 10:07:00 server named[14558]: /etc/named.conf:12: no forwarders seen; disabling forwarding
Dec 11 10:07:00 server named[14558]: set up managed keys zone for view _default, file 'managed-keys.bind'
Dec 11 10:07:10 server named[14558]: Failed to init credentials (Cannot resolve servers for KDC in realm "HUNTER.ORG")
Dec 11 10:07:10 server named[14558]: loading configuration: failure
Dec 11 10:07:10 server named[14558]: exiting (due to fatal error)
Dec 11 10:07:10 server systemd[1]: named.service: control process exited, code=exited status=1
Dec 11 10:07:10 server systemd[1]: Failed to start Berkeley Internet Name Domain (DNS).
Dec 11 10:07:10 server systemd[1]: Unit named.service entered failed state
Dec 11 10:07:10 server yum[14034]: Updated: freeipa-server-3.1.0-1.fc18.x86_64
Dec 11 10:07:25 server dbus[22839]: avc:  received policyload notice (seqno=2)
Dec 11 10:07:25 server dbus[13844]: avc:  received policyload notice (seqno=2)
Dec 11 10:07:25 server dbus[22960]: avc:  received policyload notice (seqno=2)
Dec 11 10:07:25 server dbus[628]: avc:  received policyload notice (seqno=2)
Dec 11 10:07:25 server dbus[1001]: avc:  received policyload notice (seqno=2)
Dec 11 10:07:25 server dbus[1024]: avc:  received policyload notice (seqno=2)
Dec 11 10:07:25 server dbus-daemon[628]: dbus[628]: avc:  received policyload notice (seqno=2)
Dec 11 10:07:25 server dbus-daemon[628]: dbus[628]: [system] Reloaded configuration
Dec 11 10:07:25 server dbus[628]: [system] Reloaded configuration
Dec 11 10:07:26 server yum[14034]: Updated: freeipa-server-selinux-3.1.0-1.fc18.x86_64
Dec 11 10:07:27 server yum[14034]: Updated: policycoreutils-python-2.1.13-44.fc18.x86_64
Dec 11 10:07:27 server yum[14034]: Updated: rpm-build-libs-4.10.2-1.fc18.x86_64

Comment 2 Dean Hunter 2012-12-11 16:32:28 UTC
[root@server ~]# systemctl stop ipa.service
[root@server ~]# systemctl start ipa.service
Job for ipa.service failed. See 'systemctl status ipa.service' and 'journalctl -n' for details.
[root@server ~]# systemctl status ipa.service
ipa.service - Identity, Policy, Audit
	  Loaded: loaded (/usr/lib/systemd/system/ipa.service; enabled)
	  Active: failed (Result: exit-code) since Tue, 2012-12-11 10:31:48 CST; 6s ago
	 Process: 15584 ExecStart=/usr/sbin/ipactl start (code=exited, status=1/FAILURE)
	  CGroup: name=systemd:/system/ipa.service

Dec 11 10:31:48 server ipactl[15584]: raise IpactlError("Aborting ipactl")
Dec 11 10:31:48 server ipactl[15584]: ipa: INFO: The ipactl command failed,...tl
Dec 11 10:31:48 server ipactl[15584]: Aborting ipactl
Dec 11 10:31:48 server ipactl[15584]: Starting Directory Service
Dec 11 10:31:48 server ipactl[15584]: Starting krb5kdc Service
Dec 11 10:31:48 server ipactl[15584]: Starting kadmin Service
Dec 11 10:31:48 server ipactl[15584]: Starting named Service
Dec 11 10:31:48 server systemd[1]: ipa.service: main process exited, code=e...RE
Dec 11 10:31:48 server systemd[1]: Failed to start Identity, Policy, Audit.
Dec 11 10:31:48 server systemd[1]: Unit ipa.service entered failed state
[root@server ~]#

Comment 3 Dean Hunter 2012-12-11 16:34:14 UTC
From /var/log/messages during the stop and start of ipa.service:

Dec 11 10:31:28 server systemd[1]: Stopped Identity, Policy, Audit.
Dec 11 10:31:35 server systemd[1]: Starting Identity, Policy, Audit...
Dec 11 10:31:36 server ipactl[15584]: Existing service file detected!
Dec 11 10:31:36 server ipactl[15584]: Assuming stale, cleaning and proceeding
Dec 11 10:31:36 server systemd[1]: Reached target 389 Directory Server.
Dec 11 10:31:36 server systemd[1]: Starting Kerberos 5 KDC...
Dec 11 10:31:36 server systemd[1]: Started Kerberos 5 KDC.
Dec 11 10:31:36 server systemd[1]: Starting Kerberos 5 Password-changing and Administration...
Dec 11 10:31:36 server systemd[1]: Started Kerberos 5 Password-changing and Administration.
Dec 11 10:31:36 server systemd[1]: Starting Berkeley Internet Name Domain (DNS)...
Dec 11 10:31:36 server named-checkconf[15610]: zone localhost.localdomain/IN: loaded serial 0
Dec 11 10:31:36 server named-checkconf[15610]: zone localhost/IN: loaded serial 0
Dec 11 10:31:36 server named-checkconf[15610]: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0
Dec 11 10:31:36 server named-checkconf[15610]: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0
Dec 11 10:31:36 server named-checkconf[15610]: zone 0.in-addr.arpa/IN: loaded serial 0
Dec 11 10:31:36 server named[15615]: starting BIND 9.9.2-P1-RedHat-9.9.2-5.P1.fc18 -u named
Dec 11 10:31:36 server named[15615]: built with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-libtool' '--localstatedir=/var' '--enable-threads' '--enable-ipv6' '--with-pic' '--disable-static' '--disable-openssl-version-check' '--enable-exportlib' '--with-export-libdir=/usr/lib64' '--with-export-includedir=/usr/include' '--includedir=/usr/include/bind9' '--with-pkcs11=/usr/lib64/pkcs11/PKCS11_API.so' '--with-dlz-ldap=yes' '--with-dlz-postgres=yes' '--with-dlz-mysql=yes' '--with-dlz-filesystem=yes' '--with-dlz-bdb=yes' '--with-gssapi=yes' '--disable-isc-spnego' '--enable-fixed-rrset' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro ' 'CPPFLAGS= -DDIG_SIGCHASE'
Dec 11 10:31:36 server named[15615]: ----------------------------------------------------
Dec 11 10:31:36 server named[15615]: BIND 9 is maintained by Internet Systems Consortium,
Dec 11 10:31:36 server named[15615]: Inc. (ISC), a non-profit 501(c)(3) public-benefit
Dec 11 10:31:36 server named[15615]: corporation.  Support and training for BIND 9 are
Dec 11 10:31:36 server named[15615]: available at https://www.isc.org/support
Dec 11 10:31:36 server named[15615]: ----------------------------------------------------
Dec 11 10:31:36 server named[15615]: adjusted limit on open files from 4096 to 1048576
Dec 11 10:31:36 server named[15615]: found 8 CPUs, using 8 worker threads
Dec 11 10:31:36 server named[15615]: using 8 UDP listeners per interface
Dec 11 10:31:36 server named[15615]: using up to 4096 sockets
Dec 11 10:31:36 server named[15615]: loading configuration from '/etc/named.conf'
Dec 11 10:31:36 server named[15615]: using default UDP/IPv4 port range: [1024, 65535]
Dec 11 10:31:36 server named[15615]: using default UDP/IPv6 port range: [1024, 65535]
Dec 11 10:31:36 server named[15615]: listening on IPv6 interfaces, port 53
Dec 11 10:31:36 server named[15615]: listening on IPv4 interface lo, 127.0.0.1#53
Dec 11 10:31:36 server named[15615]: listening on IPv4 interface em1, 192.168.1.11#53
Dec 11 10:31:36 server named[15615]: generating session key for dynamic DNS
Dec 11 10:31:36 server named[15615]: sizing zone task pool based on 6 zones
Dec 11 10:31:36 server named[15615]: /etc/named.conf:12: no forwarders seen; disabling forwarding
Dec 11 10:31:36 server named[15615]: set up managed keys zone for view _default, file 'managed-keys.bind'
Dec 11 10:31:47 server named[15615]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Cannot resolve servers for KDC in realm "HUNTER.ORG")
Dec 11 10:31:47 server named[15615]: bind to LDAP server failed: Local error
Dec 11 10:31:47 server named[15615]: loading configuration: failure
Dec 11 10:31:47 server named[15615]: exiting (due to fatal error)
Dec 11 10:31:47 server systemd[1]: named.service: control process exited, code=exited status=1
Dec 11 10:31:47 server systemd[1]: Failed to start Berkeley Internet Name Domain (DNS).
Dec 11 10:31:47 server systemd[1]: Unit named.service entered failed state
Dec 11 10:31:47 server ipactl[15584]: Job for named.service failed. See 'systemctl status named.service' and 'journalctl -n' for details.
Dec 11 10:31:47 server ipactl[15584]: Failed to start named Service
Dec 11 10:31:47 server ipactl[15584]: Shutting down
Dec 11 10:31:47 server systemd[1]: Stopping Kerberos 5 KDC...
Dec 11 10:31:47 server systemd[1]: Stopped Kerberos 5 KDC.
Dec 11 10:31:47 server systemd[1]: Stopping Kerberos 5 Password-changing and Administration...
Dec 11 10:31:47 server systemd[1]: kadmin.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 11 10:31:47 server systemd[1]: Stopped Kerberos 5 Password-changing and Administration.
Dec 11 10:31:47 server systemd[1]: Unit kadmin.service entered failed state
Dec 11 10:31:47 server systemd[1]: Stopped Berkeley Internet Name Domain (DNS).
Dec 11 10:31:47 server systemd[1]: Stopped IPA memcached daemon, increases IPA server performance.
Dec 11 10:31:47 server systemd[1]: Stopped The Apache HTTP Server.
Dec 11 10:31:47 server systemd[1]: Stopped target PKI Tomcat Server.
Dec 11 10:31:47 server systemd[1]: Stopping 389 Directory Server PKI-IPA....
Dec 11 10:31:47 server systemd[1]: Stopping 389 Directory Server HUNTER-ORG....
Dec 11 10:31:48 server systemd[1]: Stopped 389 Directory Server HUNTER-ORG..
Dec 11 10:31:48 server systemd[1]: Stopped 389 Directory Server PKI-IPA..
Dec 11 10:31:48 server systemd[1]: Stopping 389 Directory Server.
Dec 11 10:31:48 server systemd[1]: Stopped target 389 Directory Server.
Dec 11 10:31:48 server ipactl[15584]: ipa: INFO:   File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 617, in run_script
Dec 11 10:31:48 server ipactl[15584]: return_value = main_function()
Dec 11 10:31:48 server ipactl[15584]: File "/usr/sbin/ipactl", line 490, in main
Dec 11 10:31:48 server ipactl[15584]: ipa_start(options)
Dec 11 10:31:48 server ipactl[15584]: File "/usr/sbin/ipactl", line 261, in ipa_start
Dec 11 10:31:48 server ipactl[15584]: raise IpactlError("Aborting ipactl")
Dec 11 10:31:48 server ipactl[15584]: ipa: INFO: The ipactl command failed, exception: IpactlError: Aborting ipactl
Dec 11 10:31:48 server ipactl[15584]: Aborting ipactl
Dec 11 10:31:48 server ipactl[15584]: Starting Directory Service
Dec 11 10:31:48 server ipactl[15584]: Starting krb5kdc Service
Dec 11 10:31:48 server ipactl[15584]: Starting kadmin Service
Dec 11 10:31:48 server ipactl[15584]: Starting named Service
Dec 11 10:31:48 server systemd[1]: ipa.service: main process exited, code=exited, status=1/FAILURE
Dec 11 10:31:48 server systemd[1]: Failed to start Identity, Policy, Audit.
Dec 11 10:31:48 server systemd[1]: Unit ipa.service entered failed state

Comment 4 Rob Crittenden 2012-12-11 18:39:20 UTC
It looks like 389-ds-base isn't starting which is causing the KDC and bind to not start.

Can you look in /var/log/dirsrv/slapd-HUNTER.ORG/errors for clues?

Does this work? service dirsrv@HUNTER.ORG start

Comment 5 Dean Hunter 2012-12-11 19:00:58 UTC
[root@server ~]# systemctl start dirsrv@HUNTER.ORG.service
Job for dirsrv@HUNTER.ORG.service failed. See 'systemctl status dirsrv@HUNTER.ORG.service' and 'journalctl -n' for details.
[root@server ~]# systemctl status dirsrv@HUNTER.ORG.service
dirsrv@HUNTER.ORG.service - 389 Directory Server HUNTER.ORG.
	  Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled)
	  Active: failed (Result: resources)
	  CGroup: name=systemd:/system/dirsrv@.service/HUNTER.ORG

Dec 11 12:57:31 server systemd[1]: Starting 389 Directory Server HUNTER.ORG....
Dec 11 12:57:31 server systemd[1]: Failed to load environment files: No suc...ry
Dec 11 12:57:31 server systemd[1]: Failed to start 389 Directory Server HUN.....
Dec 11 12:57:31 server systemd[1]: Unit dirsrv@HUNTER.ORG.service entered f...te
Dec 11 12:58:12 server systemd[1]: Starting 389 Directory Server HUNTER.ORG....
Dec 11 12:58:12 server systemd[1]: Failed to load environment files: No suc...ry
Dec 11 12:58:12 server systemd[1]: Failed to start 389 Directory Server HUN.....
[root@server ~]#

Comment 6 Dean Hunter 2012-12-11 19:01:49 UTC
From /var/log/dirsrv/slapd-HUNTER.ORG/errors:

[11/Dec/2012:10:06:41 -0600] - slapd shutting down - signaling operation threads
[11/Dec/2012:10:06:41 -0600] - slapd shutting down - waiting for 28 threads to terminate
[11/Dec/2012:10:06:41 -0600] - slapd shutting down - closing down internal subsystems and plugins
[11/Dec/2012:10:06:41 -0600] - Waiting for 4 database threads to stop
[11/Dec/2012:10:06:42 -0600] - All database threads now stopped
[11/Dec/2012:10:06:42 -0600] - slapd stopped.
[11/Dec/2012:10:06:45 -0600] - 389-Directory/1.3.0.a1 B2012.284.1449 starting up
[11/Dec/2012:10:06:45 -0600] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=hunter,dc=org
[11/Dec/2012:10:06:45 -0600] schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=hunter,dc=org
[11/Dec/2012:10:06:45 -0600] schema-compat-plugin - warning: no entries set up under ou=sudoers,dc=hunter,dc=org
[11/Dec/2012:10:06:45 -0600] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=hunter,dc=org--no CoS Templates found, which should be added before the CoS Definition.
[11/Dec/2012:10:06:45 -0600] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=hunter,dc=org--no CoS Templates found, which should be added before the CoS Definition.
[11/Dec/2012:10:06:45 -0600] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[11/Dec/2012:10:06:45 -0600] - Listening on All Interfaces port 636 for LDAPS requests
[11/Dec/2012:10:06:45 -0600] - Listening on /var/run/slapd-HUNTER-ORG.socket for LDAPI requests
[11/Dec/2012:10:06:56 -0600] - slapd shutting down - signaling operation threads
[11/Dec/2012:10:06:56 -0600] - slapd shutting down - waiting for 27 threads to terminate
[11/Dec/2012:10:06:56 -0600] - slapd shutting down - closing down internal subsystems and plugins
[11/Dec/2012:10:06:56 -0600] - Waiting for 4 database threads to stop
[11/Dec/2012:10:06:57 -0600] - All database threads now stopped
[11/Dec/2012:10:06:57 -0600] - slapd stopped.
[11/Dec/2012:10:08:36 -0600] - Information: Non-Secure Port Disabled
[11/Dec/2012:10:08:36 -0600] - 389-Directory/1.3.0.a1 B2012.284.1449 starting up
[11/Dec/2012:10:08:36 -0600] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=hunter,dc=org
[11/Dec/2012:10:08:36 -0600] schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=hunter,dc=org
[11/Dec/2012:10:08:36 -0600] schema-compat-plugin - warning: no entries set up under ou=sudoers,dc=hunter,dc=org
[11/Dec/2012:10:08:36 -0600] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=hunter,dc=org--no CoS Templates found, which should be added before the CoS Definition.
[11/Dec/2012:10:08:36 -0600] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=hunter,dc=org--no CoS Templates found, which should be added before the CoS Definition.
[11/Dec/2012:10:08:37 -0600] - slapd started.  Listening on /var/run/slapd-HUNTER-ORG.socket for LDAPI requests
[11/Dec/2012:10:08:44 -0600] memberof-plugin - Memberof task starts (arg: (objectclass=*)) ...
[11/Dec/2012:10:08:44 -0600] memberof-plugin - Memberof task starts (arg: (objectclass=*)) ...
[11/Dec/2012:10:08:44 -0600] memberof-plugin - Memberof task finished (arg: (objectclass=*)) ...
[11/Dec/2012:10:08:44 -0600] memberof-plugin - Memberof task finished (arg: (objectclass=*)) ...
[11/Dec/2012:10:08:45 -0600] - slapd shutting down - signaling operation threads
[11/Dec/2012:10:08:45 -0600] - slapd shutting down - closing down internal subsystems and plugins
[11/Dec/2012:10:08:45 -0600] - Waiting for 4 database threads to stop
[11/Dec/2012:10:08:46 -0600] - All database threads now stopped
[11/Dec/2012:10:08:47 -0600] - slapd stopped.
[11/Dec/2012:10:08:48 -0600] - 389-Directory/1.3.0.a1 B2012.284.1449 starting up
[11/Dec/2012:10:08:48 -0600] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=hunter,dc=org
[11/Dec/2012:10:08:48 -0600] schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=hunter,dc=org
[11/Dec/2012:10:08:48 -0600] schema-compat-plugin - warning: no entries set up under ou=sudoers,dc=hunter,dc=org
[11/Dec/2012:10:08:48 -0600] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=hunter,dc=org--no CoS Templates found, which should be added before the CoS Definition.
[11/Dec/2012:10:08:48 -0600] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=hunter,dc=org--no CoS Templates found, which should be added before the CoS Definition.
[11/Dec/2012:10:08:48 -0600] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[11/Dec/2012:10:08:48 -0600] - Listening on All Interfaces port 636 for LDAPS requests
[11/Dec/2012:10:08:48 -0600] - Listening on /var/run/slapd-HUNTER-ORG.socket for LDAPI requests
[11/Dec/2012:10:31:47 -0600] - slapd shutting down - signaling operation threads
[11/Dec/2012:10:31:47 -0600] - slapd shutting down - waiting for 22 threads to terminate
[11/Dec/2012:10:31:47 -0600] - slapd shutting down - closing down internal subsystems and plugins
[11/Dec/2012:10:31:47 -0600] - Waiting for 4 database threads to stop
[11/Dec/2012:10:31:47 -0600] - All database threads now stopped
[11/Dec/2012:10:31:47 -0600] - slapd stopped.
[11/Dec/2012:12:57:31 -0600] - 389-Directory/1.3.0.a1 B2012.284.1449 starting up
[11/Dec/2012:12:57:31 -0600] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=hunter,dc=org
[11/Dec/2012:12:57:31 -0600] schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=hunter,dc=org
[11/Dec/2012:12:57:31 -0600] schema-compat-plugin - warning: no entries set up under ou=sudoers,dc=hunter,dc=org
[11/Dec/2012:12:57:31 -0600] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=hunter,dc=org--no CoS Templates found, which should be added before the CoS Definition.
[11/Dec/2012:12:57:31 -0600] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=hunter,dc=org--no CoS Templates found, which should be added before the CoS Definition.
[11/Dec/2012:12:57:31 -0600] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[11/Dec/2012:12:57:31 -0600] - Listening on All Interfaces port 636 for LDAPS requests
[11/Dec/2012:12:57:31 -0600] - Listening on /var/run/slapd-HUNTER-ORG.socket for LDAPI requests

Comment 7 Rob Crittenden 2012-12-11 19:39:13 UTC
That is really strange. 389-ds looks like it is starting fine but systemd is not seeing it.

Are there any AVCs being thrown by selinux?

Comment 8 Dean Hunter 2012-12-11 20:54:39 UTC
[root@server ~]# ausearch -m AVC -ts today
<no matches>
[root@server ~]#

Comment 9 Dean Hunter 2012-12-11 21:04:42 UTC
What are the environment files that failed to load:

Dec 11 12:58:12 server systemd[1]: Starting 389 Directory Server HUNTER.ORG....
Dec 11 12:58:12 server systemd[1]: Failed to load environment files: No such file or directory
Dec 11 12:58:12 server systemd[1]: dirsrv@HUNTER.ORG.service failed to run 'start' task: No such file or directory
Dec 11 12:58:12 server systemd[1]: Failed to start 389 Directory Server HUNTER.ORG..

Comment 10 Dean Hunter 2012-12-11 21:26:56 UTC
Does this help?

[root@server ~]# ls -lZ /etc/systemd/system/dirsrv*
lrwxrwxrwx. root root unconfined_u:object_r:systemd_unit_file_t:s0 dirsrv@HUNTER-ORG.service -> /lib/systemd/system/dirsrv@.service
lrwxrwxrwx. root root unconfined_u:object_r:systemd_unit_file_t:s0 dirsrv@PKI-IPA.service -> /lib/systemd/system/dirsrv@.service
[root@server ~]# 
[root@server ~]# ls -lZ /lib/systemd/system/dirsrv*
-rw-r--r--. root root system_u:object_r:systemd_unit_file_t:s0 /lib/systemd/system/dirsrv@.service
-rw-r--r--. root root system_u:object_r:systemd_unit_file_t:s0 /lib/systemd/system/dirsrv-snmp.service
-rw-r--r--. root root system_u:object_r:systemd_unit_file_t:s0 /lib/systemd/system/dirsrv.target
[root@server ~]# 
[root@server ~]# cat /lib/systemd/system/dirsrv@.service
# you usually do not want to edit this file - instead, edit the
# /etc/sysconfig/dirsrv.systemd file instead - otherwise,
# do not edit this file in /lib/systemd/system - instead, do the following:
# cp /lib/systemd/system/dirsrv\@.service /etc/systemd/system/dirsrv\@.service
# mkdir -p /etc/systemd/system/dirsrv.target.wants
# edit /etc/systemd/system/dirsrv\@.service - uncomment the LimitNOFILE=8192 line
# where %i is the name of the instance
# you may already have a symlink in
# /etc/systemd/system/dirsrv.target.wants/dirsrv@%i.service pointing to
# /lib/systemd/system/dirsrv\@.service - you will have to change it to link
# to /etc/systemd/system/dirsrv\@.service instead
# ln -s /etc/systemd/system/dirsrv\@.service /etc/systemd/system/dirsrv.target.wants/dirsrv@%i.service
# systemctl daemon-reload 
# systemctl (re)start dirsrv.target
[Unit]
Description=389 Directory Server %i.
BindTo=dirsrv.target
After=dirsrv.target

[Service]
Type=forking
EnvironmentFile=/etc/sysconfig/dirsrv
EnvironmentFile=/etc/sysconfig/dirsrv-%i
ExecStart=/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-%i -i /var/run/dirsrv/slapd-%i.pid -w /var/run/dirsrv/slapd-%i.startpid
ExecStopPost=/bin/rm -f /var/run/dirsrv/slapd-%i.pid
# if you need to set other directives e.g. LimitNOFILE=8192
# set them in this file
.include /etc/sysconfig/dirsrv.systemd
[root@server ~]# 
[root@server ~]# ls -lZ /etc/sysconfig/dirsrv*
-rw-r--r--. root   root   unconfined_u:object_r:etc_t:s0   /etc/sysconfig/dirsrv
-r--r-----. dirsrv dirsrv unconfined_u:object_r:etc_t:s0   /etc/sysconfig/dirsrv-HUNTER-ORG
-r--r-----. pkisrv dirsrv unconfined_u:object_r:etc_t:s0   /etc/sysconfig/dirsrv-PKI-IPA
-rw-r--r--. root   root   unconfined_u:object_r:etc_t:s0   /etc/sysconfig/dirsrv.systemd
-rw-r--r--. root   root   unconfined_u:object_r:etc_t:s0   /etc/sysconfig/dirsrv.systemd.orig
[root@server ~]# 
[root@server ~]# cat /etc/sysconfig/dirsrv
# This file is sourced by dirsrv upon startup to set
# the default environment for all directory server instances.
# To set instance specific defaults, use the file in the same
# directory called dirsrv-instance where "instance"
# is the name of your directory server instance e.g.
# dirsrv-localhost for the slapd-localhost instance.

# This file is in systemd EnvironmentFile format - see man systemd.exec

# In order to make more file descriptors available
# to the directory server, first make sure the system
# hard limits are raised, then use ulimit - uncomment
# out the following line and change the value to the
# desired value
# ulimit -n 8192
# note - if using systemd, ulimit won't work -  you must edit
# the systemd unit file for directory server to add the 
# LimitNOFILE option - see man systemd.exec for more info

# A per instance keytab does not make much sense for servers.
# Kerberos clients use the machine FQDN to obtain a ticket like ldap/FQDN, there
# is nothing that can make a client understand how to get a per-instance ticket.
# Therefore by default a keytab should be considered a per server option.

# Also this file is sourced for all instances, so again all
# instances would ultimately get the same keytab.

# Finally a keytab is normally named either krb5.keytab or <service>.keytab

# In order to use SASL/GSSAPI (Kerberos) the directory
# server needs to know where to find its keytab
# file - uncomment the following line and set
# the path and filename appropriately
# if using systemd, omit the "; export VARNAME" at the end

# how many seconds to wait for the startpid file to show
# up before we assume there is a problem and fail to start
# if using systemd, omit the "; export VARNAME" at the end
#STARTPID_TIME=10 ; export STARTPID_TIME
# how many seconds to wait for the pid file to show
# up before we assume there is a problem and fail to start
# if using systemd, omit the "; export VARNAME" at the end
#PID_TIME=600 ; export PID_TIME
ulimit -n 8192
KRB5_KTNAME=/etc/dirsrv/ds.keytab
export KRB5_KTNAME=/etc/dirsrv/ds.keytab
[root@server ~]# 
[root@server ~]# cat /etc/sysconfig/dirsrv-HUNTER-ORG
# This file is sourced by dirsrv upon startup to set
# the default environment for a single specific  directory
# server instances.  To set defaults for all instances, edit
# the file in the same directory called dirsrv.

# These settings are used by the start-dirsrv and
# start-slapd scripts (as well as their associates stop
# and restart scripts).  Do not edit them unless you know
# what you are doing.

# This file is in systemd EnvironmentFile format - see man systemd.exec

SERVER_DIR=/usr/lib64/dirsrv
SERVERBIN_DIR=/usr/sbin
CONFIG_DIR=/etc/dirsrv/slapd-HUNTER-ORG
INST_DIR=/var/lib/dirsrv/scripts-HUNTER-ORG
RUN_DIR=/var/run/dirsrv
DS_ROOT=
PRODUCT_NAME=slapd

# Put custom instance specific settings below here.
# if using systemd, omit the "; export VARNAME" at the end
[root@server ~]# 
[root@server ~]# cat /etc/sysconfig/dirsrv.systemd
[Service]
LimitNOFILE=8192
[root@server ~]#

Comment 11 Rich Megginson 2012-12-11 21:39:46 UTC
ugh - /etc/sysconfig/dirsrv is not in systemd format

Comment 12 Rich Megginson 2012-12-11 21:45:10 UTC
Just doing a setup and install of 389-ds-base works fine - the files are in systemd format - does ipa change these files?

Comment 13 Simo Sorce 2012-12-11 21:50:42 UTC
(In reply to comment #12)
> Just doing a setup and install of 389-ds-base works fine - the files are in
> systemd format - does ipa change these files?

Yes the FreeIPA install opens the file and adds ulimit limits as well as KRB5_KTNAME to point to the right keytab.

If this needs to change please open a freeipa ticket with details on how/where to put this information now, as well as how to know when to use the old method vs new method.

Comment 14 Rich Megginson 2012-12-11 21:57:15 UTC
(In reply to comment #13)
> (In reply to comment #12)
> > Just doing a setup and install of 389-ds-base works fine - the files are in
> > systemd format - does ipa change these files?
> 
> Yes the FreeIPA install opens the file and adds ulimit limits as well as
> KRB5_KTNAME to point to the right keytab.
> 
> If this needs to change please open a freeipa ticket with details on
> how/where to put this information now, as well as how to know when to use
> the old method vs new method.

I don't understand - I thought this had already been done and was working in F-17

Comment 15 Rich Megginson 2012-12-11 22:00:56 UTC
https://fedorahosted.org/freeipa/ticket/1990
https://fedorahosted.org/freeipa/ticket/2300
etc.
was a regression introduced with ipa 3?

Comment 16 Dean Hunter 2012-12-11 22:13:26 UTC
Please note that I was able to install, start and use IPA server on Dec 7 about 15:30. The problem was introduced on Dec 11 about 10:06 when I used yum update to get the latest updates.

[root@server ~]# ls -l /etc/sysconfig/dirsrv*
-rw-r--r--. 1 root   root   2074 Dec  7 15:33 /etc/sysconfig/dirsrv
-r--r-----. 1 dirsrv dirsrv  783 Dec  7 15:31 /etc/sysconfig/dirsrv-HUNTER-ORG
-r--r-----. 1 pkisrv dirsrv  777 Dec  7 15:30 /etc/sysconfig/dirsrv-PKI-IPA
-rw-r--r--. 1 root   root     29 Dec 11 10:06 /etc/sysconfig/dirsrv.systemd
-rw-r--r--. 1 root   root     27 Dec 11 10:06 /etc/sysconfig/dirsrv.systemd.orig
[root@server ~]#

Comment 17 Rob Crittenden 2012-12-11 22:20:47 UTC
My working F-18 install looks similar.

What is the difference between dirsrv.systemd and dirsrv.systemd.orig?

Comment 18 Dean Hunter 2012-12-11 22:32:38 UTC
They are the same:

[root@server ~]# cat /etc/sysconfig/dirsrv.systemd
[Service]
LimitNOFILE=8192
[root@server ~]#
[root@server ~]# cat /etc/sysconfig/dirsrv.systemd.orig
[Service]
LimitNOFILE=8192
[root@server ~]#

I tried commenting the [Service] line. Now the directory service starts, but there is an authentication error in the named service start:

Dec 11 16:02:15 server named[20098]: starting BIND 9.9.2-P1-RedHat-9.9.2-5.P1.fc18 -u named
Dec 11 16:02:15 server named[20098]: built with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-libtool' '--localstatedir=/var' '--enable-threads' '--enable-ipv6' '--with-pic' '--disable-static' '--disable-openssl-version-check' '--enable-exportlib' '--with-export-libdir=/usr/lib64' '--with-export-includedir=/usr/include' '--includedir=/usr/include/bind9' '--with-pkcs11=/usr/lib64/pkcs11/PKCS11_API.so' '--with-dlz-ldap=yes' '--with-dlz-postgres=yes' '--with-dlz-mysql=yes' '--with-dlz-filesystem=yes' '--with-dlz-bdb=yes' '--with-gssapi=yes' '--disable-isc-spnego' '--enable-fixed-rrset' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro ' 'CPPFLAGS= -DDIG_SIGCHASE'
Dec 11 16:02:15 server named[20098]: ----------------------------------------------------
Dec 11 16:02:15 server named[20098]: BIND 9 is maintained by Internet Systems Consortium,
Dec 11 16:02:15 server named[20098]: Inc. (ISC), a non-profit 501(c)(3) public-benefit
Dec 11 16:02:15 server named[20098]: corporation.  Support and training for BIND 9 are
Dec 11 16:02:15 server named[20098]: available at https://www.isc.org/support
Dec 11 16:02:15 server named[20098]: ----------------------------------------------------
Dec 11 16:02:15 server named[20098]: adjusted limit on open files from 4096 to 1048576
Dec 11 16:02:15 server named[20098]: found 8 CPUs, using 8 worker threads
Dec 11 16:02:15 server named[20098]: using 8 UDP listeners per interface
Dec 11 16:02:15 server named[20098]: using up to 4096 sockets
Dec 11 16:02:15 server named[20098]: loading configuration from '/etc/named.conf'
Dec 11 16:02:15 server named[20098]: using default UDP/IPv4 port range: [1024, 65535]
Dec 11 16:02:15 server named[20098]: using default UDP/IPv6 port range: [1024, 65535]
Dec 11 16:02:15 server named[20098]: listening on IPv6 interfaces, port 53
Dec 11 16:02:15 server named[20098]: listening on IPv4 interface lo, 127.0.0.1#53
Dec 11 16:02:15 server named[20098]: listening on IPv4 interface em1, 192.168.1.11#53
Dec 11 16:02:15 server named[20098]: generating session key for dynamic DNS
Dec 11 16:02:15 server named[20098]: sizing zone task pool based on 6 zones
Dec 11 16:02:15 server named[20098]: /etc/named.conf:12: no forwarders seen; disabling forwarding
Dec 11 16:02:15 server named[20098]: set up managed keys zone for view _default, file 'managed-keys.bind'
Dec 11 16:02:15 server named[20098]: bind to LDAP server failed: Authentication method not supported
Dec 11 16:02:15 server named[20098]: loading configuration: failure
Dec 11 16:02:15 server named[20098]: exiting (due to fatal error)

Comment 19 Dean Hunter 2012-12-12 06:41:57 UTC
I successfully rebuilt the server after receiving notice that an SELinux fix was available, see 885154. Please notice that /etc/sysconfig/dirsrv.* are not in systemd format.

[root@server ~]# yum list installed freeipa*
Loaded plugins: langpacks, presto, refresh-packagekit
Installed Packages
freeipa-admintools.x86_64               3.1.0-1.fc18            @updates-testing
freeipa-client.x86_64                   3.1.0-1.fc18            @updates-testing
freeipa-python.x86_64                   3.1.0-1.fc18            @updates-testing
freeipa-server.x86_64                   3.1.0-1.fc18            @updates-testing
freeipa-server-selinux.x86_64           3.1.0-1.fc18            @updates-testing
[root@server ~]# yum list installed selinux*
Loaded plugins: langpacks, presto, refresh-packagekit
Installed Packages
selinux-policy.noarch                   3.11.1-62.fc18          @updates-testing
selinux-policy-devel.noarch             3.11.1-62.fc18          @updates-testing
selinux-policy-targeted.noarch          3.11.1-62.fc18          @updates-testing
[root@server ~]# ausearch -m AVC
<no matches>
[root@server ~]# cat /etc/sysconfig/dirsrv.systemd
[Service]
LimitNOFILE=8192
[root@server ~]# cat /etc/sysconfig/dirsrv.systemd.orig
[Service]
LimitNOFILE=8192
[root@server ~]#

Comment 20 Dean Hunter 2012-12-12 13:17:06 UTC
I successfully rebuilt the client and logged-in as an IPA user. Both the client and the server have been rebuilt. Everything seems to be working, except that /etc/sysconfig/dirsrv.systemd is not in the right format.

I guess that as a user I do not care why the update failed, except that this was a rather drastic work around for the problem. But I understood that was likely when I started using beta software. That is why I have scripts to rebuild everything.

Oh, well???

Comment 21 Alexander Bokovoy 2012-12-12 13:39:37 UTC
I think you have mixed all together.

/etc/sysconfig/dirsrv and /etc/sysconfig/dirsrv-* are environmental files. They get included by systemd when EnvironmentFile directive is used. systemd ignores all statements there that do not look like variable=value, so there is no issue in its format as long as both export and variable=value lines are in place. 

(For export FOO=bar that gets translated into environment variable 'export FOO' being set to value 'bar', not a problem in our case).

/etc/sysconfig/dirsrv.systemd is included via .include directive. This file should have the same format as any service file (man systemd.unit) and it does so, including section headers.

In short, both types of files are fine.

What you see in the log is that 
Dec 11 12:58:12 server systemd[1]: Failed to load environment files: No such file or directory
Dec 11 12:58:12 server systemd[1]: dirsrv@HUNTER.ORG.service failed to run 'start' task: No such file or directory

This is different issue. I'm not sure we should not fail here (i.e. prefix file path with -) since missing /etc/sysconfig/dirsrv{,-*} means directory server instance was not configured.

Rich, if you want to proceed startup even when these configuration files are missing, feel free to change systemd service file to have
EnvironmentFile=-/etc/sysconfig/dirsrv
EnvironmentFile=-/etc/sysconfig/dirsrv-%i

Note minus preceding the path, it will make systemd happy if the file is not readable or not available at the specified path.

Comment 22 Rich Megginson 2012-12-12 14:30:29 UTC
(In reply to comment #21)
> I think you have mixed all together.
> 
> /etc/sysconfig/dirsrv and /etc/sysconfig/dirsrv-* are environmental files.
> They get included by systemd when EnvironmentFile directive is used. systemd
> ignores all statements there that do not look like variable=value, so there
> is no issue in its format as long as both export and variable=value lines
> are in place. 
> 
> (For export FOO=bar that gets translated into environment variable 'export
> FOO' being set to value 'bar', not a problem in our case).
> 
> /etc/sysconfig/dirsrv.systemd is included via .include directive. This file
> should have the same format as any service file (man systemd.unit) and it
> does so, including section headers.
> 
> In short, both types of files are fine.
> 
> What you see in the log is that 
> Dec 11 12:58:12 server systemd[1]: Failed to load environment files: No such
> file or directory
> Dec 11 12:58:12 server systemd[1]: dirsrv@HUNTER.ORG.service failed to run
> 'start' task: No such file or directory
> 
> This is different issue. I'm not sure we should not fail here (i.e. prefix
> file path with -) since missing /etc/sysconfig/dirsrv{,-*} means directory
> server instance was not configured.
> 
> Rich, if you want to proceed startup even when these configuration files are
> missing, feel free to change systemd service file to have
> EnvironmentFile=-/etc/sysconfig/dirsrv
> EnvironmentFile=-/etc/sysconfig/dirsrv-%i
> 
> Note minus preceding the path, it will make systemd happy if the file is not
> readable or not available at the specified path.

Ok - but why are they missing?

Comment 23 Alexander Bokovoy 2012-12-12 15:24:06 UTC
They probably are not missing since the logs above show their content. They might be unaccessible to the process that runs the service:

[root@server ~]# ls -lZ /etc/sysconfig/dirsrv*
-rw-r--r--. root   root   unconfined_u:object_r:etc_t:s0   /etc/sysconfig/dirsrv
-r--r-----. dirsrv dirsrv unconfined_u:object_r:etc_t:s0   /etc/sysconfig/dirsrv-HUNTER-ORG
-r--r-----. pkisrv dirsrv unconfined_u:object_r:etc_t:s0   /etc/sysconfig/dirsrv-PKI-IPA
-rw-r--r--. root   root   unconfined_u:object_r:etc_t:s0   /etc/sysconfig/dirsrv.systemd
-rw-r--r--. root   root   unconfined_u:object_r:etc_t:s0   /etc/sysconfig/dirsrv.systemd.orig

Maybe unconfined_u:object_r:etc_t:s0 is not covered by the process context?

Comment 24 Rich Megginson 2012-12-12 15:49:58 UTC
(In reply to comment #23)
> They probably are not missing since the logs above show their content. They
> might be unaccessible to the process that runs the service:
> 
> [root@server ~]# ls -lZ /etc/sysconfig/dirsrv*
> -rw-r--r--. root   root   unconfined_u:object_r:etc_t:s0  
> /etc/sysconfig/dirsrv
> -r--r-----. dirsrv dirsrv unconfined_u:object_r:etc_t:s0  
> /etc/sysconfig/dirsrv-HUNTER-ORG
> -r--r-----. pkisrv dirsrv unconfined_u:object_r:etc_t:s0  
> /etc/sysconfig/dirsrv-PKI-IPA
> -rw-r--r--. root   root   unconfined_u:object_r:etc_t:s0  
> /etc/sysconfig/dirsrv.systemd
> -rw-r--r--. root   root   unconfined_u:object_r:etc_t:s0  
> /etc/sysconfig/dirsrv.systemd.orig
> 
> Maybe unconfined_u:object_r:etc_t:s0 is not covered by the process context?

Check for AVCs.

At any rate, if this is the case, I do not want to allow the server to start if these files cannot be read, unless IPA absolutely requires the server to start, even though it may not work correctly due to these missing files.

For example, the directory server gets the kerberos keytab by the KRB5_KTNAME setting in /etc/sysconfig/dirsrv - so if this file cannot be read, the directory server will not be able to use kerberos at all, for incoming as well as outgoing (replication).

Comment 25 Alexander Bokovoy 2012-12-12 15:58:23 UTC
Comment 8 says there were no AVCs.

> At any rate, if this is the case, I do not want to allow the server to start 
> if these files cannot be read, unless IPA absolutely requires the server to 
> start, even though it may not work correctly due to these missing files.
Yep, this is why I'm also hesitant to put them there.

Dean,

did you do anything specific in this VM prior to installing FreeIPA?

Comment 26 Dean Hunter 2012-12-12 16:44:27 UTC
I am using physical machines, one for the server and one for the client. I have enough trouble with real boxes without adding virtual trouble. ;-)

There was one work-around required for an SELinux problem that has since been fixed, see 885154, and another work-around for a directory that the installation was not creating, see 880995.

Here is the script I followed to install Free IPA Server 3.0.1-1:

# Install Gnome Desktop from CD and create Local Administrator during first boot

# Configure System Settings
#   Background
#     Colors & Gradients: solid dark gray
#   Network
#     Wireless: Off
#     Wired: On
#       IPv4 Settings
#         Method: Manual
#         Addresses
#           Address: 192.168.1.11
#           Netmask: 255.255.255.0
#           Gateway: 192.168.1.1
#         DNS servers: 192.168.1.11, 75.75.76.76, 75.75.75.75
#       IPv6 Settings
#         Method: Ignore
#   Sound
#     Output
#       Output volume: 100%
#     Sound Effects
#       Alert volume: Off

# Perform these steps as root to customize the install.

  su -   # requests root password

# Set the host name, because the install does not

  hostnamectl set-hostname server

  cat >/etc/sysconfig/network <<EOD
NETWORKING=yes
HOSTNAME=server.hunter.org
EOD

# Enable persistent storage for the systemd journal

  mkdir /var/log/journal

# Shutdown services that are not needed

  chkconfig iscsi        off   # Internet SCSI service
  chkconfig iscsid       off   # Internet SCSI daemon

# Install additional packages for configuration

  yum install --assumeyes dconf-editor gnome-tweak-tool system-config-lvm

# Adjust configuration using dconf-editor
#   org
#     gnome
#       shell
#         always-show-log-out: yes

  dconf-editor

# Apply queued updates

  yum update --assumeyes

# Restart to implement changes and updates

  reboot



# Perform these steps as root to customize the install.

  su -   # requests root password

# Install and configure IPA server

  yum install --assumeyes freeipa-server bind bind-dyndb-ldap

  mkdir /var/run/ipa                             # 880995
  chmod 0700 /var/run/ipa                        # 880995

  audit2allow -M mypol <<EOD                     # 885154
type=AVC msg=audit(1354894518.888:342): avc:  denied  { name_bind } for  pid=2924 comm="ns-slapd" src=7389 scontext=system_u:system_r:dirsrv_t:s0 tcontext=system_u:object_r:pki_ca_port_t:s0 tclass=tcp_socket


type=SYSCALL msg=audit(1354894518.888:342): arch=x86_64 syscall=bind success=no exit=EACCES a0=6 a1=7fff3c600d50 a2=1c a3=7fff3c600ae0 items=0 ppid=1 pid=2924 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=ns-slapd exe=/usr/sbin/ns-slapd subj=system_u:system_r:dirsrv_t:s0 key=(null)
EOD
  semodule -i mypol.pp                           # 885154

  cat >>/etc/hosts <<EOD
192.168.1.11  server.hunter.org server
EOD

  ipa-server-install \
    --admin-password adminpassword \
    --hostname server.hunter.org \
    --domain hunter.org \
    --ds-password dspassword \
    --realm HUNTER.ORG \
    --setup-dns \
    --no-forwarders \
    --unattended

  #  tcp   udp  service   description
  # ----  ----  --------  --------------
  #   53    53  dns       Domain Name Server
  #   80        http      World Wide Web HTTP
  #   88    88  kerberos  Kerberos
  #        123  ntp       Network Time Protocol
  #  389        ldap      Lightweight Directory Access Protocol
  #  443        https     World Wide Web HTTP over SSL
  #  464   464  kpasswd   kpasswd
  #  636        ldaps     Lightweight Directory Access Protocol over SSL

  firewall-config

  kinit admin   # Requests admin password

  ipa dnsrecord-add hunter.org ipa --cname-rec server

  ipa config-mod --defaultshell /bin/bash
  ipa user-add dean \
    --first Dean \
    --last Hunter \
    --password   # Requests initial password for user

# Configure authentication
#   Identity & Authentication
#     User Account Configuration
#       User Account Database: FreeIPA
#       LDAP Search Base DN: dc=hunter, dc=org
#       LDAP Server: ldaps://server.hunter.org
#       Use TLS to encrypt connections: No          (default)
#     Authentification Configuration
#       Realm: HUNTER.ORG                           (default)
#       KDCs:                                       (default)
#       Admin                                       (default)
#       Use DNS to resolve host to realms: No       (default)
#       Use DNS to locate KDCs for realms: Yes      (default)
#   Advanced Options
#     Local Authentication Options
#       Enable fingerprint reader support: No
#       Enable Local access control: No             (default)
#       Password Hashing Algorithm: SHA512          (default)
#     Other Authentication Options
#       Create home directories on the first login: Yes
#     Smart Card Authentication Options
#       Enable smart card support: No               (default)
#   Password Options                                (default)

I used the same script without the work-around for 885154 last to successfully rebuild with 3.1.0-1.


Note You need to log in before you can comment on or make changes to this bug.