Bug 1319912

Summary: ipa-server-install does not completely change hostname and named-pkcs11 fails
Product: Red Hat Enterprise Linux 7 Reporter: Scott Poore <spoore>
Component: ipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: Kaleem <ksiddiqu>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.2CC: mbasti, pspacek, pvoborni, rcritten
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ipa-4.4.0-0.el7.1.alpha1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 05:52:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Scott Poore 2016-03-21 19:25:12 UTC
Description of problem:

Using hostname/ip-address args with ipa-server-install fails if new hostname added to /etc/hosts.  The failure seen during install is:

Done configuring DNS key synchronization service (ipa-dnskeysyncd).
Restarting ipa-dnskeysyncd
Restarting named
ipa         : ERROR    Named service failed to start (Command ''/bin/systemctl' 'restart' 'named-pkcs11.service'' returned non-zero exit status 1)
named service failed to start
Restarting the web server
ipa.ipapython.install.cli.install_tool(Server): ERROR    Command ''/bin/systemctl' 'restart' 'ipa.service'' returned non-zero exit status 1

In /var/log/messages I see references to EXAMPLE.COM:

Mar 21 13:09:00 vm1 named-pkcs11[12688]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/EXAMPLE.COM not found in Kerberos database)

Mar 21 13:09:00 vm1 named-pkcs11[12688]: LDAP error: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/EXAMPLE.COM not found in Kerberos database): bind to LDAP server failed


Version-Release number of selected component (if applicable):
ipa-server-4.2.0-15.el7_2.6.x86_64
also seen in:
ipa-server-4.2.0-15.el7_2.12.x86_64

How reproducible:
Unknown but has been seen in a couple different test environments.

Steps to Reproduce:
1.  yum install ipa-server-dns
2.  echo "$(hostname -i) master.testrelm.test" >> /etc/hosts
3.  ipa-server-install --setup-dns --forwarder=$FORWARDER --hostname=master.testrelm.test --ip-address=$(hostname -i) -n testrelm.test -r TESTRELM.TEST -a Secret123 -p Secret123 -U

Actual results:

Fails as shown above.

Expected results:

No failure. Changes hostname everywhere expected.

Additional info:

Comment 1 Scott Poore 2016-03-21 19:25:58 UTC
Also note:

[root@vm1 ~]# hostname
vm1.example.com

Comment 2 Scott Poore 2016-03-21 19:27:24 UTC
FYI both of these seem to work around this:

1. remove entry from /etc/hosts and rely on ipa-server-install to add that.

2. fully change hostname in /etc/hostname and if necessary with hostname command before running ipa-server-install.

Comment 4 Petr Vobornik 2016-03-24 19:13:55 UTC
Petr, I believe we've seen this bug before but couldn't reproduce. Could you look at it?

Comment 5 Scott Poore 2016-03-24 19:21:53 UTC
Petr Vobornik, 

Are you talking about this one?

bug #1247247

Or maybe a different one?  That's the only one like it that I ran into when I was looking at this.  I wasn't 100% sure if the same or different.  If we need to close this and re-open that one, we can.

Thanks,
Scott

Comment 6 Petr Spacek 2016-03-24 19:22:49 UTC
I do not see how this can happen, it is most likely some weird misconfiguration related to run-time state of kernel or so. IMHO it is a waste of time unless there is a clear reproducer -> INSUFFICIENT_DATA.

Comment 7 Petr Spacek 2016-03-24 19:24:28 UTC
BTW I do not see a reason why the hell ipa-server-install should change system's hostname. That should be done by provisioning system before the installer is executed.

Comment 8 Rob Crittenden 2016-03-24 19:44:27 UTC
(In reply to Petr Spacek from comment #7)
> BTW I do not see a reason why the hell ipa-server-install should change
> system's hostname. That should be done by provisioning system before the
> installer is executed.

What provisioning system is that?

This is particularly useful in the multi-NIC case so the right hostname gets set.

The --hostname option came about because so many users could not seem to get their hostname right (it would be a short hostname or would be completely wrong). Is it IdM's problem to solve? It is when it gets the blame for things not working.

Comment 9 Petr Vobornik 2016-03-24 19:55:51 UTC
Yes Scott, I think it was bug 1247247.

Comment 10 Petr Spacek 2016-03-29 07:30:07 UTC
(In reply to Rob Crittenden from comment #8)
> (In reply to Petr Spacek from comment #7)
> > BTW I do not see a reason why the hell ipa-server-install should change
> > system's hostname. That should be done by provisioning system before the
> > installer is executed.
> 
> What provisioning system is that?
That is up to the user. IPA is not here to dictate how things should be provisioned. It can be kickstart generated by Satellite/Cobbler or even manual clicking in Anaconda if user wishes.

> This is particularly useful in the multi-NIC case so the right hostname gets
> set.
I do not understand this point. This is definition of struct uname in Linux:
           struct utsname {
               char sysname[];    /* Operating system name (e.g., "Linux") */
               char nodename[];   /* Name within "some implementation-defined
                                     network" */
               char release[];    /* Operating system release (e.g., "2.6.28") */
               char version[];    /* Operating system version */
               char machine[];    /* Hardware identifier */
           #ifdef _GNU_SOURCE
               char domainname[]; /* NIS or YP domain name */
           #endif
           };

There is only 1 nodename (often wrongly aliased to hostname) per system so there is simply nothing to select. If you are talking about PTR record lookups from IP addresses to names, well, no option in IPA CLI will make a difference as long as records in DNS do not match values configured on the system.


> The --hostname option came about because so many users could not seem to get
> their hostname right (it would be a short hostname or would be completely
> wrong). Is it IdM's problem to solve? It is when it gets the blame for
> things not working.
Usually the main problem is debugability. We should have error message which tells exactly what is wrong and what is the expected state so it is easy to fix that.

I do not see how --hostname option solves the problem that user is attempting to configure IPA on system where nodename = short name. If he tries that an error message should be printed but this should be no different from running installer on system with short name already configured.

Comment 11 Rob Crittenden 2016-03-29 13:06:48 UTC
(In reply to Petr Spacek from comment #10)
> (In reply to Rob Crittenden from comment #8)
> > (In reply to Petr Spacek from comment #7)
> > > BTW I do not see a reason why the hell ipa-server-install should change
> > > system's hostname. That should be done by provisioning system before the
> > > installer is executed.
> > 
> > What provisioning system is that?
> That is up to the user. IPA is not here to dictate how things should be
> provisioned. It can be kickstart generated by Satellite/Cobbler or even
> manual clicking in Anaconda if user wishes.

My point is that more often than not this is some guy at a keyboard.
 
> > This is particularly useful in the multi-NIC case so the right hostname gets
> > set.
> I do not understand this point. This is definition of struct uname in Linux:
...
> There is only 1 nodename (often wrongly aliased to hostname) per system so
> there is simply nothing to select. If you are talking about PTR record
> lookups from IP addresses to names, well, no option in IPA CLI will make a
> difference as long as records in DNS do not match values configured on the
> system.

Many libraries use the system hostname so which NIC gets the system name can be important otherwise the wrong name will be picked when attempting to use a service.
 
> I do not see how --hostname option solves the problem that user is
> attempting to configure IPA on system where nodename = short name. If he
> tries that an error message should be printed but this should be no
> different from running installer on system with short name already
> configured.

The option dates back to very early days of the project. I believe that updating the system hostname came later. In either case, it is a way to tell an installer to configure things using a given name rather than socket.getfqdn().

Comment 12 Petr Vobornik 2016-03-29 17:14:43 UTC
What is the expected behavior of --hostname option?

Always set static fqdn? 

If so then we can set it regardless of result from hostname resolution.

From man page it seems that it should be set only if the hostname differs. Such behavior is odd. It basically says it might or not be set static, currently depends on network setup.

Regardless of it, I don't know why checking current hostname depends on socket.getfqdn(). If we want to prevent using shortnames, then socket.gethostname() is better(assuming that it returns nodename).

Network verification should be done when installing replica but in server-install it should be just consistency check.


Btw, we may even replace in backup_and_replace_hostname:
 * # hostname $hostname
 * write $hostname into /etc/hostname

with single step:
 * # hostnamectl set-hostname $hostname

Comment 13 Martin Bašti 2016-03-30 11:34:49 UTC
I agree. We should always set hostname by hostnamectl if --hostname option is specified and do not do any validation (except shortname) in installers.

Comment 14 Petr Spacek 2016-03-31 08:12:13 UTC
I think that validation is important piece of installers. If you insist on keeping the option, okay but it should be moved to platform specific code as this might potentially differ between distros.

Comment 15 Petr Vobornik 2016-04-08 08:52:07 UTC
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/5794

Comment 18 Scott Poore 2016-07-15 02:33:05 UTC
Verified.

Version ::

ipa-server-dns-4.4.0-2.1.el7.noarch

Results ::

[root@rhel7-1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@rhel7-1 ~]# cat /etc/hostname
rhel7-1.example.com
[root@rhel7-1 ~]# echo "192.168.122.71 master.testrelm.test" >> /etc/hosts
[root@rhel7-1 ~]# ipa-server-install --setup-dns --forwarder=192.168.122.1 --reverse-zone=122.168.192.in-addr.arp. --allow-zone-overlap --hostname=master.testrelm.test --ip-address=192.168.122.71 --domain=testrelm.test --realm=TESTRELM.TEST --ds-password=Secret123 --admin-password=Secret123 -U

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the Network Time Daemon (ntpd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)

WARNING: conflicting time&date synchronization service 'chronyd' will be disabled
in favor of ntpd

Warning: skipping DNS resolution of host master.testrelm.test
Checking DNS domain testrelm.test., please wait ...
Checking DNS forwarders, please wait ...
Using reverse zone(s) 122.168.192.in-addr.arp.

The IPA Master Server will be configured with:
Hostname:       master.testrelm.test
IP address(es): 192.168.122.71
Domain name:    testrelm.test
Realm name:     TESTRELM.TEST

BIND DNS server will be configured to serve IPA domain with:
Forwarders:       192.168.122.1
Forward policy:   only
Reverse zone(s):  122.168.192.in-addr.arp.

Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 1 minute
  [1/47]: creating directory server user
  [2/47]: creating directory server instance
  [3/47]: updating configuration in dse.ldif
  [4/47]: restarting directory server
  [5/47]: adding default schema
  [6/47]: enabling memberof plugin
  [7/47]: enabling winsync plugin
  [8/47]: configuring replication version plugin
  [9/47]: enabling IPA enrollment plugin
  [10/47]: enabling ldapi
  [11/47]: configuring uniqueness plugin
  [12/47]: configuring uuid plugin
  [13/47]: configuring modrdn plugin
  [14/47]: configuring DNS plugin
  [15/47]: enabling entryUSN plugin
  [16/47]: configuring lockout plugin
  [17/47]: configuring topology plugin
  [18/47]: creating indices
  [19/47]: enabling referential integrity plugin
  [20/47]: configuring certmap.conf
  [21/47]: configure autobind for root
  [22/47]: configure new location for managed entries
  [23/47]: configure dirsrv ccache
  [24/47]: enabling SASL mapping fallback
  [25/47]: restarting directory server
  [26/47]: adding sasl mappings to the directory
  [27/47]: adding default layout
  [28/47]: adding delegation layout
  [29/47]: creating container for managed entries
  [30/47]: configuring user private groups
  [31/47]: configuring netgroups from hostgroups
  [32/47]: creating default Sudo bind user
  [33/47]: creating default Auto Member layout
  [34/47]: adding range check plugin
  [35/47]: creating default HBAC rule allow_all
  [36/47]: adding sasl mappings to the directory
  [37/47]: adding entries for topology management
  [38/47]: initializing group membership
  [39/47]: adding master entry
  [40/47]: initializing domain level
  [41/47]: configuring Posix uid/gid generation
  [42/47]: adding replication acis
  [43/47]: enabling compatibility plugin
  [44/47]: activating sidgen plugin
  [45/47]: activating extdom plugin
  [46/47]: tuning directory server
  [47/47]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds
  [1/31]: creating certificate server user
  [2/31]: configuring certificate server instance
  [3/31]: stopping certificate server instance to update CS.cfg
  [4/31]: backing up CS.cfg
  [5/31]: disabling nonces
  [6/31]: set up CRL publishing
  [7/31]: enable PKIX certificate path discovery and validation
  [8/31]: starting certificate server instance
  [9/31]: creating RA agent certificate database
  [10/31]: importing CA chain to RA certificate database
  [11/31]: fixing RA database permissions
  [12/31]: setting up signing cert profile
  [13/31]: setting audit signing renewal to 2 years
  [14/31]: restarting certificate server
  [15/31]: requesting RA certificate from CA
  [16/31]: issuing RA agent certificate
  [17/31]: adding RA agent as a trusted user
  [18/31]: authorizing RA to modify profiles
  [19/31]: authorizing RA to manage lightweight CAs
  [20/31]: Ensure lightweight CAs container exists
  [21/31]: configure certmonger for renewals
  [22/31]: configure certificate renewals
  [23/31]: configure RA certificate renewal
  [24/31]: configure Server-Cert certificate renewal
  [25/31]: Configure HTTP to proxy connections
  [26/31]: restarting certificate server
  [27/31]: migrating certificate profiles to LDAP
  [28/31]: importing IPA certificate profiles
  [29/31]: adding default CA ACL
  [30/31]: adding 'ipa' CA entry
  [31/31]: updating IPA configuration
Done configuring certificate server (pki-tomcatd).
Configuring directory server (dirsrv). Estimated time: 10 seconds
  [1/3]: configuring ssl for ds instance
  [2/3]: restarting directory server
  [3/3]: adding CA certificate entry
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc). Estimated time: 30 seconds
  [1/9]: adding kerberos container to the directory
  [2/9]: configuring KDC
  [3/9]: initialize kerberos container
  [4/9]: adding default ACIs
  [5/9]: creating a keytab for the directory
  [6/9]: creating a keytab for the machine
  [7/9]: adding the password extension to the directory
  [8/9]: starting the KDC
  [9/9]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring ipa_memcached
  [1/2]: starting ipa_memcached 
  [2/2]: configuring ipa_memcached to start on boot
Done configuring ipa_memcached.
Configuring ipa-otpd
  [1/2]: starting ipa-otpd 
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Configuring ipa-custodia
  [1/5]: Generating ipa-custodia config file
  [2/5]: Making sure custodia container exists
  [3/5]: Generating ipa-custodia keys
  [4/5]: starting ipa-custodia 
  [5/5]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Configuring the web interface (httpd). Estimated time: 1 minute
  [1/21]: setting mod_nss port to 443
  [2/21]: setting mod_nss cipher suite
  [3/21]: setting mod_nss protocol list to TLSv1.0 - TLSv1.2
  [4/21]: setting mod_nss password file
  [5/21]: enabling mod_nss renegotiate
  [6/21]: adding URL rewriting rules
  [7/21]: configuring httpd
  [8/21]: configure certmonger for renewals
  [9/21]: setting up httpd keytab
  [10/21]: setting up ssl
  [11/21]: importing CA certificates from LDAP
  [12/21]: setting up browser autoconfig
  [13/21]: publish CA cert
  [14/21]: clean up any existing httpd ccache
  [15/21]: configuring SELinux for httpd
  [16/21]: create KDC proxy user
  [17/21]: create KDC proxy config
  [18/21]: enable KDC proxy
  [19/21]: restarting httpd
  [20/21]: configuring httpd to start on boot
  [21/21]: enabling oddjobd
Done configuring the web interface (httpd).
Applying LDAP updates
Upgrading IPA:
  [1/9]: stopping directory server
  [2/9]: saving configuration
  [3/9]: disabling listeners
  [4/9]: enabling DS global lock
  [5/9]: starting directory server
  [6/9]: upgrading server
  [7/9]: stopping directory server
  [8/9]: restoring configuration
  [9/9]: starting directory server
Done.
Restarting the directory server
Restarting the KDC
Configuring DNS (named)
  [1/12]: generating rndc key file
  [2/12]: adding DNS container
  [3/12]: setting up our zone
  [4/12]: setting up reverse zone
  [5/12]: setting up our own record
  [6/12]: setting up records for other masters
  [7/12]: adding NS record to the zones
  [8/12]: setting up kerberos principal
  [9/12]: setting up named.conf
  [10/12]: setting up server configuration
  [11/12]: configuring named to start on boot
  [12/12]: changing resolv.conf to point to ourselves
Done configuring DNS (named).
Configuring DNS key synchronization service (ipa-dnskeysyncd)
  [1/7]: checking status
  [2/7]: setting up bind-dyndb-ldap working directory
  [3/7]: setting up kerberos principal
  [4/7]: setting up SoftHSM
  [5/7]: adding DNSSEC containers
  [6/7]: creating replica keys
  [7/7]: configuring ipa-dnskeysyncd to start on boot
Done configuring DNS key synchronization service (ipa-dnskeysyncd).
Restarting ipa-dnskeysyncd
Restarting named
Updating DNS system records
Restarting the web server
Configuring client side components
Using existing certificate '/etc/ipa/ca.crt'.
Client hostname: master.testrelm.test
Realm: TESTRELM.TEST
DNS Domain: testrelm.test
IPA Server: master.testrelm.test
BaseDN: dc=testrelm,dc=test

Skipping synchronizing time with NTP server.
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
trying https://master.testrelm.test/ipa/json
Forwarding 'schema' to json server 'https://master.testrelm.test/ipa/json'
trying https://master.testrelm.test/ipa/json
Forwarding 'ping' to json server 'https://master.testrelm.test/ipa/json'
Forwarding 'ca_is_enabled' to json server 'https://master.testrelm.test/ipa/json'
Systemwide CA database updated.
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Forwarding 'host_mod' to json server 'https://master.testrelm.test/ipa/json'
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring testrelm.test as NIS domain.
Client configuration complete.

==============================================================================
Setup complete

Next steps:
	1. You must make sure these network ports are open:
		TCP Ports:
		  * 80, 443: HTTP/HTTPS
		  * 389, 636: LDAP/LDAPS
		  * 88, 464: kerberos
		  * 53: bind
		UDP Ports:
		  * 88, 464: kerberos
		  * 53: bind
		  * 123: ntp

	2. You can now obtain a kerberos ticket using the command: 'kinit admin'
	   This ticket will allow you to use the IPA tools (e.g., ipa user-add)
	   and the web user interface.

Be sure to back up the CA certificates stored in /root/cacert.p12
These files are required to create replicas. The password for these
files is the Directory Manager password

[root@rhel7-1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.122.71 master.testrelm.test

[root@rhel7-1 ~]# cat /etc/hostname
master.testrelm.test

[root@rhel7-1 ~]# hostname
master.testrelm.test

[root@rhel7-1 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

Comment 20 errata-xmlrpc 2016-11-04 05:52:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2404.html