Bug 1534462

Summary: kinit command doesnot work post ipa-server install
Product: Red Hat Enterprise Linux 7 Reporter: Sudhir Menon <sumenon>
Component: 389-ds-baseAssignee: mreynolds
Status: CLOSED ERRATA QA Contact: Viktor Ashirov <vashirov>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.5CC: ksiddiqu, lslebodn, nkinder, pvoborni, rcritten, rmeggins, slaznick, spoore, tbordaz, tscherf, vashirov
Target Milestone: rcKeywords: Regression, TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.3.7.5-13 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 14:23:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
dirsrv logs
none
ipa-server install log
none
dirsrv errors log with nsslapd-errorlog-level=5 none

Description Sudhir Menon 2018-01-15 10:04:24 UTC 
Description of problem: kinit doesn't work post ipa-server install
 

Version-Release number of selected component (if applicable):
RHEL-7.5-20180111.1 Server x86_64 
ipa-server-4.5.4-8.el7.x86_64
389-ds-base-1.3.7.5-12.el7.x86_64

How reproducible: Always


Steps to Reproduce:
1. Install IPA server. Ensure installation is successful.
2. kinit admin


Actual results:
[root@master slapd-SDR2K16-TEST]# kinit admin
kinit: Pre-authentication failed: Invalid argument while getting initial credentials

Expected results:
kinit should work


Additional info: Attaching the dirsrv error logs.
[15/Jan/2018:04:18:56.693046487 -0500] - ERR - connection_threadmain - pb_conn is NULL
[15/Jan/2018:04:18:56.694680085 -0500] - ERR - connection_threadmain - pb_conn is NULL
[15/Jan/2018:04:18:56.696657294 -0500] - ERR - connection_threadmain - pb_conn is NULL
[15/Jan/2018:04:18:56.698658849 -0500] - ERR - connection_threadmain - pb_conn is NULL
[15/Jan/2018:04:18:56.699618998 -0500] - ERR - connection_threadmain - pb_conn is NULL
[15/Jan/2018:04:18:56.709335567 -0500] - ERR - connection_threadmain - pb_conn is NULL

ns-slapd: [15/Jan/2018:04:19:12.733390361 -0500] - ERR - op_shared_allow_pw_change - NULL param error: conn (0x(nil)) op (0x0x556e0721e8c0)
Comment 2 Sudhir Menon 2018-01-15 10:06:17 UTC 
Created attachment 1381342 [details]
dirsrv logs
Comment 3 Sudhir Menon 2018-01-15 10:10:25 UTC 
Created attachment 1381343 [details]
ipa-server install log
Comment 5 Sudhir Menon 2018-01-15 10:21:42 UTC 
[root@master ~]# env KRB5_TRACE=/dev/stdout kinit admin
[11721] 1516011360.245522: Getting initial credentials for admin
[11721] 1516011360.245524: Sending request (173 bytes) to SDR2K16.TEST
[11721] 1516011360.245525: Initiating TCP connection to stream 
[11721] 1516011360.245526: Sending TCP request to stream 
[11721] 1516011360.245527: Received answer (249 bytes) from stream 
[11721] 1516011360.245528: Terminating TCP connection to stream
[11721] 1516011360.245529: Response was from master KDC
[11721] 1516011360.245530: Received error from KDC: -1765328359/Additional pre-authentication required
[11721] 1516011360.245533: Processing preauth types: 16, 15, 14, 136, 147, 133
[11721] 1516011360.245534: Received cookie: MIT
[11721] 1516011360.245535: PKINIT client has no configured identity; giving up
[11721] 1516011360.245536: Preauth module pkinit (147) (info) returned: 0/Success
[11721] 1516011360.245537: PKINIT client has no configured identity; giving up
[11721] 1516011360.245538: Preauth module pkinit (16) (real) returned: 22/Invalid argument
[11721] 1516011360.245539: PKINIT client has no configured identity; giving up
[11721] 1516011360.245540: Preauth module pkinit (14) (real) returned: 22/Invalid argument
[11721] 1516011360.245541: PKINIT client has no configured identity; giving up
[11721] 1516011360.245542: Preauth module pkinit (14) (real) returned: 22/Invalid argument
kinit: Pre-authentication failed: Invalid argument while getting initial credentials

Note: this works with the earlier version of 389-ds-base
ipa-server-4.5.4-8.el7.x86_64
389-ds-base-libs-1.3.7.5-11.el7.x86_64
Comment 6 thierry bordaz 2018-01-15 10:55:44 UTC 
    - The error is coming from an internal mod triggered by ipa-pwd-extop (on 1.3.6.1.4.1.4203.1.11.1 extop)
[15/Jan/2018:04:19:12.733948383 -0500] - ERR - op_shared_allow_pw_change - NULL param error: conn (0x(nil)) op (0x0x556e0721e8c0)

[15/Jan/2018:04:19:12.676980344 -0500] conn=18 fd=96 slot=96 connection from ...
[15/Jan/2018:04:19:12.677163026 -0500] conn=18 op=0 EXT oid="1.3.6.1.4.1.1466.20037" name="start_tls_plugin"
[15/Jan/2018:04:19:12.677199068 -0500] conn=18 op=0 RESULT err=0 tag=120 nentries=0 etime=0.0000191802
[15/Jan/2018:04:19:12.719266161 -0500] conn=18 TLS1.2 256-bit AES-GCM
[15/Jan/2018:04:19:12.719574264 -0500] conn=18 op=1 BIND dn="cn=Directory Manager" method=128 version=3
[15/Jan/2018:04:19:12.719672470 -0500] conn=18 op=1 RESULT err=0 tag=97 nentries=0 etime=0.0006416006 dn="cn=directory manager"
[15/Jan/2018:04:19:12.719811057 -0500] conn=18 op=2 EXT oid="1.3.6.1.4.1.4203.1.11.1" name="IPA Password Manager"
[15/Jan/2018:04:19:12.736777214 -0500] conn=18 op=2 RESULT err=0 tag=120 nentries=0 etime=0.0017026392
[15/Jan/2018:04:19:12.740682655 -0500] conn=18 op=3 UNBIND
[15/Jan/2018:04:19:12.740697755 -0500] conn=18 op=3 fd=96 closed - U1

    - This error prevents updating the password and further kinit

    - The error looks related to a recent 389-ds fix 
        https://pagure.io/389-ds-base/issue/49529
        https://bugzilla.redhat.com/show_bug.cgi?id=1519406
      but surprised to see it in RHEL

In conclusion, RC is possibly a side effect 49529 but need to confirm it exists in 389-ds-base-1.3.7.5-12.el7.x86_64
Comment 7 thierry bordaz 2018-01-15 11:11:57 UTC 
Indeed, #49529 / bz 1519406 are fixed in 389-ds-base-1.3.7.5-12.el7.x86_64.
The problem is that the coverity fix does not take into account internal operation where all pblock fields are not set (like connection). It erroneously reject the internal MOD.

This is a 389-ds bug, changing the component.
Comment 8 mreynolds 2018-01-15 13:56:35 UTC 
Any chance we can see a stack trace leading up the erro message?

The coverity fix did address a potential crash in op_shared_allow_pw_change(), but if there are cases where pb_conn is legitimately NULL I need to know them because there are a lot of NULL deferences possible if pb_conn is NULL, and also if the pblock operation is NULL too.
Comment 9 Scott Poore 2018-01-15 23:50:58 UTC 
Created attachment 1381774 [details]
dirsrv errors log with nsslapd-errorlog-level=5

Does the attached errors file provide the trace you're looking for?

I set logging by running:


# ldapmodify -x -D "cn=Directory Manager" -w Secret123 <<EOF
> dn: cn=config
> changetype: modify
> replace: nsslapd-errorlog-level
> nsslapd-errorlog-level: 5
> EOF
modifying entry "cn=config"
Comment 11 Lukas Slebodnik 2018-01-16 09:45:24 UTC 
(In reply to thierry bordaz from comment #7)
> Indeed, #49529 / bz 1519406 are fixed in 389-ds-base-1.3.7.5-12.el7.x86_64.
> The problem is that the coverity fix does not take into account internal
> operation where all pblock fields are not set (like connection). It
> erroneously reject the internal MOD.
> 
> This is a 389-ds bug, changing the component.

I can confirm that removing patch 0052-Ticket-49529-Fix-Coverity-warnings-invalid-deference.patch fixed issue with "kinit admin"
Comment 12 mreynolds 2018-01-16 14:30:35 UTC 
The regression was identified and it is going to be checked in and fixed in a build later today.
Comment 16 Scott Poore 2018-01-19 16:46:34 UTC 
I can confirm that this fixed the kinit issue after a fresh install:

[root@rhel7-3 ~]# rpm -q 389-ds-base
389-ds-base-1.3.7.5-13.el7.x86_64


[root@rhel7-3 ~]# hostnamectl set-hostname rhel7-3.testrelm.test 
[root@rhel7-3 ~]# ipa-server-install --setup-dns --forwarder=192.168.122.1 --auto-reverse -n testrelm.test -r TESTRELM.TEST -a Secret123 -p Secret123 -U
Checking DNS domain testrelm.test, please wait ...

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the Network Time Daemon (ntpd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)
  * Configure the KDC to enable PKINIT

WARNING: conflicting time&date synchronization service 'chronyd' will be disabled
in favor of ntpd

Warning: skipping DNS resolution of host rhel7-3.testrelm.test
Checking DNS domain testrelm.test., please wait ...
Checking DNS forwarders, please wait ...

The IPA Master Server will be configured with:
Hostname:       rhel7-3.testrelm.test
IP address(es): 192.168.122.73
Domain name:    testrelm.test
Realm name:     TESTRELM.TEST

BIND DNS server will be configured to serve IPA domain with:
Forwarders:       192.168.122.1
Forward policy:   only
Reverse zone(s):  No reverse zone

Adding [192.168.122.73 rhel7-3.testrelm.test] to your /etc/hosts file
Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/45]: creating directory server instance
  [2/45]: enabling ldapi
  [3/45]: configure autobind for root
  [4/45]: stopping directory server
  [5/45]: updating configuration in dse.ldif
  [6/45]: starting directory server
  [7/45]: adding default schema
  [8/45]: enabling memberof plugin
  [9/45]: enabling winsync plugin
  [10/45]: configuring replication version plugin
  [11/45]: enabling IPA enrollment plugin
  [12/45]: configuring uniqueness plugin
  [13/45]: configuring uuid plugin
  [14/45]: configuring modrdn plugin
  [15/45]: configuring DNS plugin
  [16/45]: enabling entryUSN plugin
  [17/45]: configuring lockout plugin
  [18/45]: configuring topology plugin
  [19/45]: creating indices
  [20/45]: enabling referential integrity plugin
  [21/45]: configuring certmap.conf
  [22/45]: configure new location for managed entries
  [23/45]: configure dirsrv ccache
  [24/45]: enabling SASL mapping fallback
  [25/45]: restarting directory server
  [26/45]: adding sasl mappings to the directory
  [27/45]: adding default layout
  [28/45]: adding delegation layout
  [29/45]: creating container for managed entries
  [30/45]: configuring user private groups
  [31/45]: configuring netgroups from hostgroups
  [32/45]: creating default Sudo bind user
  [33/45]: creating default Auto Member layout
  [34/45]: adding range check plugin
  [35/45]: creating default HBAC rule allow_all
  [36/45]: adding entries for topology management
  [37/45]: initializing group membership
  [38/45]: adding master entry
  [39/45]: initializing domain level
  [40/45]: configuring Posix uid/gid generation
  [41/45]: adding replication acis
  [42/45]: activating sidgen plugin
  [43/45]: activating extdom plugin
  [44/45]: tuning directory server
  [45/45]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc)
  [1/10]: adding kerberos container to the directory
  [2/10]: configuring KDC
  [3/10]: initialize kerberos container
  [4/10]: adding default ACIs
  [5/10]: creating a keytab for the directory
  [6/10]: creating a keytab for the machine
  [7/10]: adding the password extension to the directory
  [8/10]: creating anonymous principal
  [9/10]: starting the KDC
  [10/10]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [4/29]: backing up CS.cfg
  [5/29]: disabling nonces
  [6/29]: set up CRL publishing
  [7/29]: enable PKIX certificate path discovery and validation
  [8/29]: starting certificate server instance
  [9/29]: configure certmonger for renewals
  [10/29]: requesting RA certificate from CA
  [11/29]: setting up signing cert profile
  [12/29]: setting audit signing renewal to 2 years
  [13/29]: restarting certificate server
  [14/29]: publishing the CA certificate
  [15/29]: adding RA agent as a trusted user
  [16/29]: authorizing RA to modify profiles
  [17/29]: authorizing RA to manage lightweight CAs
  [18/29]: Ensure lightweight CAs container exists
  [19/29]: configure certificate renewals
  [20/29]: configure Server-Cert certificate renewal
  [21/29]: Configure HTTP to proxy connections
  [22/29]: restarting certificate server
  [23/29]: updating IPA configuration
  [24/29]: enabling CA instance
  [25/29]: migrating certificate profiles to LDAP
  [26/29]: importing IPA certificate profiles
  [27/29]: adding default CA ACL
  [28/29]: adding 'ipa' CA entry
  [29/29]: configuring certmonger renewal for lightweight CAs
Done configuring certificate server (pki-tomcatd).
Configuring directory server (dirsrv)
  [1/3]: configuring TLS for DS instance
  [2/3]: adding CA certificate entry
  [3/3]: restarting directory server
Done configuring directory server (dirsrv).
Configuring ipa-otpd
  [1/2]: starting ipa-otpd 
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Configuring ipa-custodia
  [1/5]: Generating ipa-custodia config file
  [2/5]: Making sure custodia container exists
  [3/5]: Generating ipa-custodia keys
  [4/5]: starting ipa-custodia 
  [5/5]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Configuring the web interface (httpd)
  [1/22]: stopping httpd
  [2/22]: setting mod_nss port to 443
  [3/22]: setting mod_nss cipher suite
  [4/22]: setting mod_nss protocol list to TLSv1.0 - TLSv1.2
  [5/22]: setting mod_nss password file
  [6/22]: enabling mod_nss renegotiate
  [7/22]: disabling mod_nss OCSP
  [8/22]: adding URL rewriting rules
  [9/22]: configuring httpd
  [10/22]: setting up httpd keytab
  [11/22]: configuring Gssproxy
  [12/22]: setting up ssl
  [13/22]: configure certmonger for renewals
  [14/22]: importing CA certificates from LDAP
  [15/22]: publish CA cert
  [16/22]: clean up any existing httpd ccaches
  [17/22]: configuring SELinux for httpd
  [18/22]: create KDC proxy config
  [19/22]: enable KDC proxy
  [20/22]: starting httpd
  [21/22]: configuring httpd to start on boot
  [22/22]: enabling oddjobd
Done configuring the web interface (httpd).
Configuring Kerberos KDC (krb5kdc)
  [1/1]: installing X509 Certificate for PKINIT
Done configuring Kerberos KDC (krb5kdc).
Applying LDAP updates
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/9]: stopping directory server
  [2/9]: saving configuration
  [3/9]: disabling listeners
  [4/9]: enabling DS global lock
  [5/9]: starting directory server
  [6/9]: upgrading server
  [7/9]: stopping directory server
  [8/9]: restoring configuration
  [9/9]: starting directory server
Done.
Restarting the KDC
Configuring DNS (named)
  [1/11]: generating rndc key file
  [2/11]: adding DNS container
  [3/11]: setting up our zone
  [4/11]: setting up our own record
  [5/11]: setting up records for other masters
  [6/11]: adding NS record to the zones
  [7/11]: setting up kerberos principal
  [8/11]: setting up named.conf
  [9/11]: setting up server configuration
  [10/11]: configuring named to start on boot
  [11/11]: changing resolv.conf to point to ourselves
Done configuring DNS (named).
Restarting the web server to pick up resolv.conf changes
Configuring DNS key synchronization service (ipa-dnskeysyncd)
  [1/7]: checking status
  [2/7]: setting up bind-dyndb-ldap working directory
  [3/7]: setting up kerberos principal
  [4/7]: setting up SoftHSM
  [5/7]: adding DNSSEC containers
  [6/7]: creating replica keys
  [7/7]: configuring ipa-dnskeysyncd to start on boot
Done configuring DNS key synchronization service (ipa-dnskeysyncd).
Restarting ipa-dnskeysyncd
Restarting named
Updating DNS system records
Configuring client side components
Using existing certificate '/etc/ipa/ca.crt'.
Client hostname: rhel7-3.testrelm.test
Realm: TESTRELM.TEST
DNS Domain: testrelm.test
IPA Server: rhel7-3.testrelm.test
BaseDN: dc=testrelm,dc=test

Skipping synchronizing time with NTP server.
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
trying https://rhel7-3.testrelm.test/ipa/json
[try 1]: Forwarding 'schema' to json server 'https://rhel7-3.testrelm.test/ipa/json'
trying https://rhel7-3.testrelm.test/ipa/session/json
[try 1]: Forwarding 'ping' to json server 'https://rhel7-3.testrelm.test/ipa/session/json'
[try 1]: Forwarding 'ca_is_enabled' to json server 'https://rhel7-3.testrelm.test/ipa/session/json'
Systemwide CA database updated.
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
[try 1]: Forwarding 'host_mod' to json server 'https://rhel7-3.testrelm.test/ipa/session/json'
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring testrelm.test as NIS domain.
Client configuration complete.
The ipa-client-install command was successful

==============================================================================
Setup complete

Next steps:
	1. You must make sure these network ports are open:
		TCP Ports:
		  * 80, 443: HTTP/HTTPS
		  * 389, 636: LDAP/LDAPS
		  * 88, 464: kerberos
		  * 53: bind
		UDP Ports:
		  * 88, 464: kerberos
		  * 53: bind
		  * 123: ntp

	2. You can now obtain a kerberos ticket using the command: 'kinit admin'
	   This ticket will allow you to use the IPA tools (e.g., ipa user-add)
	   and the web user interface.

Be sure to back up the CA certificates stored in /root/cacert.p12
These files are required to create replicas. The password for these
files is the Directory Manager password


[root@rhel7-3 ~]# kinit admin
Password for admin:
Comment 19 Scott Poore 2018-01-19 17:07:41 UTC 
Verified in comment #16
Comment 22 errata-xmlrpc 2018-04-10 14:23:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0811