Bug 1449238

Summary: ipa-server install command fails
Product: Red Hat Enterprise Linux 7 Reporter: Nikhil Dehadrai <ndehadra>
Component: gssproxyAssignee: Robbie Harwood <rharwood>
Status: CLOSED ERRATA QA Contact: Nikhil Dehadrai <ndehadra>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 7.4CC: ajmitchell, chunwang, dpal, eguan, fs-qe, ksiddiqu, ndehadra, nsoman, pvoborni, rcritten, rharwood, ssorce, swhiteho, tscherf, xdong, yoyang
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: gssproxy-0.7.0-4.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 20:55:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1452881    
Bug Blocks:    
Attachments:
Description Flags
Console Output none

Description Nikhil Dehadrai 2017-05-09 13:15:44 UTC
Description of problem:
ipa-server install command fails with "CalledProcessError: Command '/bin/systemctl restart gssproxy.service' returned non-zero exit status 1" error.

Version-Release number of selected component (if applicable):
ipa-server-4.5.0-10.el7.x86_64:

How reproducible:


Environment:(In my case)
1. VM
2. RAM: 4GB
3. CPU: 8

Steps to Reproduce:
1. Setup RHEL 7.4 on VM system.
2. Configure latest repo links to it.
3. Run the following command:
# ipa-server-install --ip-address x.x.x.x -r testrelm.test -p 'Secret123' -a 'Secret123' --forwarder x.x.x.x --setup-dns -U


Actual results:
The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.
 
This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the Network Time Daemon (ntpd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)
  * Configure the KDC to enable PKINIT
 
WARNING: conflicting time&date synchronization service 'chronyd' will be disabled
in favor of ntpd
 
Warning: skipping DNS resolution of host dhcp213-66.testrelm.test
The domain name has been determined based on the host name.
 
Checking DNS domain testrelm.test., please wait ...
Checking DNS forwarders, please wait ...
 
The IPA Master Server will be configured with:
Hostname:       IPAMASTER.testrelm.test
IP address(es): x.x.x.x
Domain name:    testrelm.test
Realm name:     TESTRELM.TEST
 
BIND DNS server will be configured to serve IPA domain with:
Forwarders:       x.x.x.x
Forward policy:   only
Reverse zone(s):  No reverse zone
 
Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/45]: creating directory server instance
  [2/45]: enabling ldapi
  [3/45]: configure autobind for root
  [4/45]: stopping directory server
  [5/45]: updating configuration in dse.ldif
  [6/45]: starting directory server
  [7/45]: adding default schema
  [8/45]: enabling memberof plugin
  [9/45]: enabling winsync plugin
  [10/45]: configuring replication version plugin
  [11/45]: enabling IPA enrollment plugin
  [12/45]: configuring uniqueness plugin
  [13/45]: configuring uuid plugin
  [14/45]: configuring modrdn plugin
  [15/45]: configuring DNS plugin
  [16/45]: enabling entryUSN plugin
  [17/45]: configuring lockout plugin
  [18/45]: configuring topology plugin
  [19/45]: creating indices
  [20/45]: enabling referential integrity plugin
  [21/45]: configuring certmap.conf
  [22/45]: configure new location for managed entries
  [23/45]: configure dirsrv ccache
  [24/45]: enabling SASL mapping fallback
  [25/45]: restarting directory server
  [26/45]: adding sasl mappings to the directory
  [27/45]: adding default layout
  [28/45]: adding delegation layout
  [29/45]: creating container for managed entries
  [30/45]: configuring user private groups
  [31/45]: configuring netgroups from hostgroups
  [32/45]: creating default Sudo bind user
  [33/45]: creating default Auto Member layout
  [34/45]: adding range check plugin
  [35/45]: creating default HBAC rule allow_all
  [36/45]: adding entries for topology management
  [37/45]: initializing group membership
  [38/45]: adding master entry
  [39/45]: initializing domain level
  [40/45]: configuring Posix uid/gid generation
  [41/45]: adding replication acis
  [42/45]: activating sidgen plugin
  [43/45]: activating extdom plugin
  [44/45]: tuning directory server
  [45/45]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc)
  [1/10]: adding kerberos container to the directory
  [2/10]: configuring KDC
  [3/10]: initialize kerberos container
  [4/10]: adding default ACIs
  [5/10]: creating a keytab for the directory
  [6/10]: creating a keytab for the machine
  [7/10]: adding the password extension to the directory
  [8/10]: creating anonymous principal
  [9/10]: starting the KDC
  [10/10]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [4/29]: backing up CS.cfg
  [5/29]: disabling nonces
  [6/29]: set up CRL publishing
  [7/29]: enable PKIX certificate path discovery and validation
  [8/29]: starting certificate server instance
  [9/29]: configure certmonger for renewals
  [10/29]: requesting RA certificate from CA
  [11/29]: setting up signing cert profile
  [12/29]: setting audit signing renewal to 2 years
  [13/29]: restarting certificate server
  [14/29]: publishing the CA certificate
  [15/29]: adding RA agent as a trusted user
  [16/29]: authorizing RA to modify profiles
  [17/29]: authorizing RA to manage lightweight CAs
  [18/29]: Ensure lightweight CAs container exists
  [19/29]: configure certificate renewals
  [20/29]: configure Server-Cert certificate renewal
  [21/29]: Configure HTTP to proxy connections
  [22/29]: restarting certificate server
  [23/29]: updating IPA configuration
  [24/29]: enabling CA instance
  [25/29]: migrating certificate profiles to LDAP
  [26/29]: importing IPA certificate profiles
  [27/29]: adding default CA ACL
  [28/29]: adding 'ipa' CA entry
  [29/29]: configuring certmonger renewal for lightweight CAs
Done configuring certificate server (pki-tomcatd).
Configuring directory server (dirsrv)
  [1/3]: configuring TLS for DS instance
  [2/3]: adding CA certificate entry
  [3/3]: restarting directory server
Done configuring directory server (dirsrv).
Configuring ipa-otpd
  [1/2]: starting ipa-otpd
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Configuring ipa-custodia
  [1/5]: Generating ipa-custodia config file
  [2/5]: Making sure custodia container exists
  [3/5]: Generating ipa-custodia keys
  [4/5]: starting ipa-custodia
  [5/5]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Configuring the web interface (httpd)
  [1/21]: stopping httpd
  [2/21]: setting mod_nss port to 443
  [3/21]: setting mod_nss cipher suite
  [4/21]: setting mod_nss protocol list to TLSv1.0 - TLSv1.2
  [5/21]: setting mod_nss password file
  [6/21]: enabling mod_nss renegotiate
  [7/21]: adding URL rewriting rules
  [8/21]: configuring httpd
  [9/21]: setting up httpd keytab
  [10/21]: configuring Gssproxy
  [error] CalledProcessError: Command '/bin/systemctl restart gssproxy.service' returned non-zero exit status 1
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    Command '/bin/systemctl restart gssproxy.service' returned non-zero exit status 1
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information


Expected results:
ipa server install command should be successful.

Additional info:
Installation on another system with regular hardware specs RAM 2GB, it went successful.

Comment 7 Robbie Harwood 2017-05-09 18:02:58 UTC
Apologies, I see that you have.  You need to reboot after upgrading the kernel/installing nfs-utils before this will work.

You can alternately uninstall nfs-utils.

If you consider this behavior incorrect, please retarget the bug to nfs-utils.

Comment 8 Xiyang Dong 2017-05-09 19:41:02 UTC
In our case, where we run tests in CI, nfs-utils got installed after kernel got updated.
Moving this to nfs-utils

Comment 9 Steve Dickson 2017-05-09 19:55:17 UTC
(In reply to Xiyang Dong from comment #8)
> In our case, where we run tests in CI, nfs-utils got installed after kernel
> got updated.
> Moving this to nfs-utils

Why is this an nfs-utils problem?

Comment 10 Xiyang Dong 2017-05-09 20:18:41 UTC
Hi Robbie, could you please explain why nfs-utils can't work with gssproxy after kernel update? Thanks

Comment 11 Robbie Harwood 2017-05-09 20:36:31 UTC
You need to restart the machine after installing nfs-utils before the socket will be exposed for gssproxy.  As for why this is, I don't know; you need to ask nfs-utils that.

Comment 14 Steve Whitehouse 2017-05-17 11:04:52 UTC
Does that mean that this issue is resolved, or is there something reproducible that we can investigate? I'm not sure that I fully follow the above comments, so perhaps someone can summarize where we are with this one at the moment?

Comment 15 Alice Mitchell 2017-05-18 09:02:14 UTC
I can reproduce this on 7.4-beta, and it only happens the first time you run ipa-server-install immediately after installing the nfs-utils rpm. 

If the system is rebooted, or gssproxy is restarted manually with systemctl between the installation of nfs-utils and the running of ipa-server-install then this bug does not appear.

Comment 16 Alice Mitchell 2017-05-19 16:44:15 UTC
This smells like a systemd dependancy issue, but i haven't quite figured out where yet. The nub of the issue is that at the point where gssproxy is being started by ipa-server-install using systemctl the kernel module auth_rpcgss is not already loaded, and thus the access of /proc/net/rpc/gss-use-proxy fails.

if you performed the same command from the cmdline then a dependancy causes the kernel module to be loaded and everything succeeds.  and as above if there has been a reboot, or any other action that would have caused knfsd and the other modules to load, then the problem does not appear.

Comment 17 Simo Sorce 2017-05-19 18:20:21 UTC
I think we addressed this start problem in gss-proxy recently, but not sure it is in 7.4

Comment 18 Robbie Harwood 2017-05-19 18:59:28 UTC
(In reply to Simo Sorce from comment #17)
> I think we addressed this start problem in gss-proxy recently, but not sure
> it is in 7.4

If nfs-utils is installed, then the snippet is present, and gssproxy will try to use the proc file.  If nfs-utils somehow isn't set up, there isn't really anything gssproxy can do about it.

Comment 19 Steve Dickson 2017-05-22 17:47:55 UTC
(In reply to Robbie Harwood from comment #18)
> (In reply to Simo Sorce from comment #17)
> > I think we addressed this start problem in gss-proxy recently, but not sure
> > it is in 7.4
> 
> If nfs-utils is installed, then the snippet is present, and gssproxy will
> try to use the proc file.  If nfs-utils somehow isn't set up, there isn't
> really anything gssproxy can do about it.

I'm thinking the restart done by bug 1440887 probably
took care of this problem...

Comment 20 Alice Mitchell 2017-05-23 14:08:54 UTC
The addition of 'Wants: auth-rpcgss-module' to gssproxy.service seems to fix this, as the dependencies are then satisfied, but does doing that cause any knock on effects ?

Comment 21 Alice Mitchell 2017-05-25 14:21:43 UTC
gssproxy-0.7.0-7 when built for rhel74 does appear to fix this

Comment 22 Robbie Harwood 2017-05-25 16:35:45 UTC
(In reply to Justin Mitchell from comment #20)
> The addition of 'Wants: auth-rpcgss-module' to gssproxy.service seems to fix
> this, as the dependencies are then satisfied, but does doing that cause any
> knock on effects ?

gssproxy can't depend on nfs-utils, and to my understanding, that would cause gssproxy to fail to start when nfs-utils isn't installed.

Comment 23 Alice Mitchell 2017-05-26 12:01:19 UTC
I was mistaken gssproxy-0.7.0-7 did not fix the fault, it only appeared to due to a bug in its spec file, the %triggerun section goes and deletes /etc/gssproxy/24-nfs-server.conf even if nfs-utils is still installed, such that when gssproxy is later started it then skips the check that should have failed.

The patch posted on bug 1452881 does appear to work, at least ipa-server-install no longer fails.

Comment 24 Robbie Harwood 2017-05-31 17:49:17 UTC
Per steved's request, gssproxy will work around this behavior.

Comment 26 Nikhil Dehadrai 2017-06-06 09:29:11 UTC
IPA: ipa-server-4.5.0-14.el7.x86_64
GSSPROXY: gssproxy-0.7.0-4.el7.x86_64

Verified that:
1. IPA-MASTER is successfully installed on system with 4GB RAM and 8CPU.
2. No error message is observed during installation.

Refer attached log.

Thus on the basis of above observations, marking status of bug to "VERIFIED".

Comment 27 Nikhil Dehadrai 2017-06-06 09:29:40 UTC
Created attachment 1285310 [details]
Console Output

Console Output

Comment 28 errata-xmlrpc 2017-08-01 20:55:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2033