Bug 1486225

Upstream ticket:
https://pagure.io/freeipa/issue/7087

A bug has been filed against NSS to allow it to read PKCS#12 files from older NSS versions:

  https://bugzilla.redhat.com/show_bug.cgi?id=1491444

In addition to fixing NSS, we should fix Dogtag (pki-core) to generate PKCS#12 files in the correct format for newer NSS versions (and for openssl).  Moving this bug to the pki-core component for this part of the problem.

Upstream pki ticket: https://pagure.io/dogtagpki/issue/2809
Gerrit review: https://review.gerrithub.io/#/c/378753/1

edewata reviewed and pushed ftweedal changes:

Author: Fraser Tweedale <ftweedal>
Date:   Thu Sep 14 12:22:47 2017 +1000

    Make PKCS #12 files compatible with OpenSSL, NSS >= 3.31
    
    For compatibility with OpenSSL and NSS >= 3.31, the passphrase must
    not be BMPString-encoded when non-PKCS #12 PBE schemes such as
    PBES2.
    
    Fixes: https://pagure.io/dogtagpki/issue/2809
    
    Change-Id: Ic78ad337ac0b9b2f5d2e75581cc0ee55e6d82782

NOTE: edewata believes that we still need to update the pki-core.spec
      file to require the appropriate version of NSS (e. g. - >= 3.31 on
      Fedora, and hopefully nss >= 3.28-? on RHEL once the proper changes
      have been cherry-picked to RHEL)

We don't need to update the spec file in relation to this change.  This change
makes Dogtag compatible with older and newer NSS.

But I think we do need to update spec file in relation to a different
JSS change, oncen the relevant builds/releases of JSS are available.

Ade Lee 2017-10-26 13:26:50 EDT

Commits (in order):

386357c347f8433e14ccd8637576f4c4a4e42492
bc329a0162ae9af382c81e75742b282ea8c5df0d
9eb354883c9d965bb271223bf870839bb756db26
fa2d731b6ce51c5db9fb0b004d586b8f3e1decd3
8c0a7eee3bbfe01b2d965dbe09e95221c5031c8b

IPA-server version: ipa-server-4.5.4-7.el7.x86_64

Tested the bug with following observations:
1. Setup IPA master with KRA at domain Level 0.
2. Prepare Replica file for Replica.
3. Setup Replica with kra against ipa-master at domain level0.

Observations:
1. After step1, ipa-master is installed successfully
ipa-server-install --setup-dns --forwarder <forwarder_ip> --domain testrelm.test --realm TESTRELM.TEST --admin-password <admin-password> --ds-password <ds-password> -U --reverse-zone <reverse_zone_details> --allow-zone-overlap --setup-kra --domain-level=0

2. After step3, the replica installation fails:
#  ipa-replica-install -U --setup-dns --forwarder <forwarder_ip> --setup-ca --setup-kra --admin-password <admin-password> --password <password> /var/lib/ipa/replica-info-replica.testrelm.test.gpg

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/28]: configuring certificate server instance
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmp7HAvcR' returned non-zero exit status 1
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information:
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    CA configuration failed.
ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

Thus on the basis of above observations, changing status of bug to "ASSIGNED"

Nikhil, what were the expact package versions of ipa and pki-core used for
the test?

(In reply to Fraser Tweedale from comment #17)
> Nikhil, what were the expact package versions of ipa and pki-core used for
> the test?

# rpm -q ipa-server pki-core
ipa-server-4.5.4-7.el7.x86_64
package pki-core is not installed

# rpm -qa | grep pki
pki-server-10.5.1-5.el7.noarch
pki-base-java-10.5.1-5.el7.noarch
pki-ca-10.5.1-5.el7.noarch
krb5-pkinit-1.15.1-18.el7.x86_64
pki-base-10.5.1-5.el7.noarch
pki-tools-10.5.1-5.el7.x86_64
pki-kra-10.5.1-5.el7.noarch

Let me know if you need anymore information.

Confirmed that the nss version in use (nss-3.34.0-3.el7.x86_64)
contains the password encoding fix.

The fix for https://pagure.io/dogtagpki/issue/2557 is also included
in pki 10.5.1-5.el7.

The debug log shows errors in setting up replication:

[09/Jan/2018:06:47:46][http-bio-8443-exec-3]: setupReplication: consumer initialization failed. -11  - LDAP error: Connect error
[09/Jan/2018:06:47:46][http-bio-8443-exec-3]: setupReplication: java.io.IOException: consumer initialization failed. -11  - LDAP error: Connect error
java.io.IOException: Failed to setup the replication for cloning.
	at com.netscape.cms.servlet.csadmin.ConfigurationUtils.setupReplication(ConfigurationUtils.java:2070)
	at org.dogtagpki.server.rest.SystemConfigService.initializeDatabase(SystemConfigService.java:713)
	at org.dogtagpki.server.ca.rest.CAInstallerService.initializeDatabase(CAInstallerService.java:116)

I haven't encountered this kind of error before.  It is not related to the fixes previously
associated with this ticket and may be a regression elsewhere.

Is there anything of note in the DS error or access log?

Will continue investigations tomorrow.

(In reply to Fraser Tweedale from comment #19)
> Confirmed that the nss version in use (nss-3.34.0-3.el7.x86_64)
> contains the password encoding fix.
> 
> The fix for https://pagure.io/dogtagpki/issue/2557 is also included
> in pki 10.5.1-5.el7.
> 
> The debug log shows errors in setting up replication:
> 
> [09/Jan/2018:06:47:46][http-bio-8443-exec-3]: setupReplication: consumer
> initialization failed. -11  - LDAP error: Connect error
> [09/Jan/2018:06:47:46][http-bio-8443-exec-3]: setupReplication:
> java.io.IOException: consumer initialization failed. -11  - LDAP error:
> Connect error
> java.io.IOException: Failed to setup the replication for cloning.
> 	at
> com.netscape.cms.servlet.csadmin.ConfigurationUtils.
> setupReplication(ConfigurationUtils.java:2070)
> 	at
> org.dogtagpki.server.rest.SystemConfigService.
> initializeDatabase(SystemConfigService.java:713)
> 	at
> org.dogtagpki.server.ca.rest.CAInstallerService.
> initializeDatabase(CAInstallerService.java:116)
> 
> I haven't encountered this kind of error before.  It is not related to the
> fixes previously
> associated with this ticket and may be a regression elsewhere.
> 
> Is there anything of note in the DS error or access log?
> 
> Will continue investigations tomorrow.


Hi Fraser,

This is what I could get from DB error logs:

[09/Jan/2018:06:44:09.658701995 -0500] - INFO - Security Initialization - SSL info:     TLS_AES_256_GCM_SHA384: enabled
[09/Jan/2018:06:44:09.698924923 -0500] - INFO - Security Initialization - slapd_ssl_init2 - Configured SSL version range: min: TLS1.0, max: TLS1.2
[09/Jan/2018:06:44:09.708657456 -0500] - INFO - main - 389-Directory/1.3.7.5 B2017.342.2028 starting up
[09/Jan/2018:06:44:17.192386262 -0500] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
[09/Jan/2018:06:44:17.231422793 -0500] - WARN - default_mr_indexer_create - Plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match
[09/Jan/2018:06:44:17.242546217 -0500] - NOTICE - ldbm_back_start - found 1833416k physical memory
[09/Jan/2018:06:44:17.244622311 -0500] - NOTICE - ldbm_back_start - found 1265760k available
[09/Jan/2018:06:44:17.246620969 -0500] - NOTICE - ldbm_back_start - cache autosizing: db cache: 45835k
[09/Jan/2018:06:44:17.249095195 -0500] - NOTICE - ldbm_back_start - cache autosizing: userRoot entry cache (1 total): 131072k
[09/Jan/2018:06:44:17.253804567 -0500] - NOTICE - ldbm_back_start - cache autosizing: userRoot dn cache (1 total): 65536k
[09/Jan/2018:06:44:17.257579538 -0500] - NOTICE - ldbm_back_start - total cache size: 238874951 B; 
[09/Jan/2018:06:44:17.484824656 -0500] - ERR - attrcrypt_cipher_init - No symmetric key found for cipher AES in backend userRoot, attempting to create one...
[09/Jan/2018:06:44:17.502005411 -0500] - INFO - attrcrypt_cipher_init - Key for cipher AES successfully generated and stored
[09/Jan/2018:06:44:17.504378128 -0500] - ERR - attrcrypt_cipher_init - No symmetric key found for cipher 3DES in backend userRoot, attempting to create one...
[09/Jan/2018:06:44:17.519884184 -0500] - INFO - attrcrypt_cipher_init - Key for cipher 3DES successfully generated and stored
[09/Jan/2018:06:44:17.575003965 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=groups,cn=compat,dc=testrelm,dc=test does not exist
[09/Jan/2018:06:44:17.578635413 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=computers,cn=compat,dc=testrelm,dc=test does not exist
[09/Jan/2018:06:44:17.580847379 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=ng,cn=compat,dc=testrelm,dc=test does not exist
[09/Jan/2018:06:44:17.583613869 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target ou=sudoers,dc=testrelm,dc=test does not exist
[09/Jan/2018:06:44:17.587267755 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=users,cn=compat,dc=testrelm,dc=test does not exist
[09/Jan/2018:06:44:17.595150396 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=ad,cn=etc,dc=testrelm,dc=test does not exist
[09/Jan/2018:06:44:17.620153340 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=testrelm,dc=test does not exist
[09/Jan/2018:06:44:17.622421384 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=testrelm,dc=test does not exist
[09/Jan/2018:06:44:17.829732545 -0500] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=testrelm,dc=test--no CoS Templates found, which should be added before the CoS Definition.
[09/Jan/2018:06:44:17.908908578 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipaqavmb.testrelm.test] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))
[09/Jan/2018:06:44:17.931425822 -0500] - INFO - slapd_daemon - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[09/Jan/2018:06:44:17.933635954 -0500] - INFO - slapd_daemon - Listening on All Interfaces port 636 for LDAPS requests
[09/Jan/2018:06:44:17.935733007 -0500] - INFO - slapd_daemon - Listening on /var/run/slapd-TESTRELM-TEST.socket for LDAPI requests
[09/Jan/2018:06:45:54.435254800 -0500] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
[09/Jan/2018:06:45:54.519814883 -0500] - ERR - attrcrypt_cipher_init - No symmetric key found for cipher AES in backend ipaca, attempting to create one...
[09/Jan/2018:06:45:54.545681336 -0500] - INFO - attrcrypt_cipher_init - Key for cipher AES successfully generated and stored
[09/Jan/2018:06:45:54.555292711 -0500] - ERR - attrcrypt_cipher_init - No symmetric key found for cipher 3DES in backend ipaca, attempting to create one...
[09/Jan/2018:06:45:54.587747960 -0500] - INFO - attrcrypt_cipher_init - Key for cipher 3DES successfully generated and stored
[09/Jan/2018:06:45:54.628389194 -0500] - ERR - ipa-topology-plugin - ipa_topo_be_state_change - backend ipaca is coming online; checking domain level and init shared topology
[09/Jan/2018:06:45:54.678544386 -0500] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=testrelm,dc=test--no CoS Templates found, which should be added before the CoS Definition.
[09/Jan/2018:06:47:30.774289178 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -11 (Connect error)
[09/Jan/2018:06:47:30.778253927 -0500] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=cloneAgreement1-ipaqavmb.testrelm.test-pki-tomcat" (auto-hv-01-guest10:389) - Replication bind with SIMPLE auth failed: LDAP error -11 (Connect error) (error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (self signed certificate in certificate chain))
[09/Jan/2018:06:47:33.804309057 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -11 (Connect error)
[09/Jan/2018:06:47:39.826847902 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -11 (Connect error)
[09/Jan/2018:06:47:51.968539276 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -11 (Connect error)
[09/Jan/2018:06:48:15.022542284 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -11 (Connect error)


Nothing in access log during that instance.

Same or similar errors - a failure to establish replication - occurs when
performing DL0 CA installation either as part of ipa-replica-install (with or without --setup-kra) or in ipa-ca-install after performing replica installation without `--setup-ca'.  So the problem does not seem to involve the KRA specifically.

Does not occur in DL1.  One difference between replica installation on DL0 vs DL1 is that in DL0 Dogtag itself sets up the replication as part of pkispawn, whereas in DL1 the database is created and replication is configured before invoking pkispawn.

Specifically, in the ipa-server-install --setup-ca case, the DS error log shows TLS handshake failures ("self signed certificate in chain").  In the ipa-ca-install case, it shows "Unable to get certificate CRL".  AFAICT both master and replica DS NSSDBs contain the CA certificate with correct trust flags.

Feeling quite out of my depth here, and I would appreciate if someone who understands how DS replication is established could take a look at it.

I'll continue investigation tomorrow.

Hoo boy, this is a curly one.

So, I think it *may* be related to a change in openldap-2.4.44.9:

  * Fri Nov 03 2017 Matus Honek <mhonek> - 2.4.44-9 - Build with 
    OpenSSL and MozNSS compatibility layer instead of MozNSS (#1400578)

This occurs between RHEL 7.4 and RHEL 7.5.  Notably, this change does not seem to have occurred in Fedora.

Now, when 389ds detects that the OpenLDAP client is using the OpenSSL backend,
it sets LDAP_CRLCHECK option to `all'.  In setup_ol_tls_conn(LDAP *ld, int clientauth):

        if (slapi_client_uses_openssl(ld)) {
            const int crlcheck = LDAP_OPT_X_TLS_CRL_ALL;
            /* Sets the CRL evaluation strategy. */
            rc = ldap_set_option(ld, LDAP_OPT_X_TLS_CRLCHECK, &crlcheck);
            if (rc) {
                slapi_log_err(SLAPI_LOG_ERR, "setup_ol_tls_conn",
                              "Could not set CRLCHECK [%d]: %d:%s\n",
                              crlcheck, rc, ldap_err2string(rc));
            }
        }

Replication setup uses these methods to communicate between replicas.  I don't fully understand how replication gets set up but will discuss with wibrown tomorrow.  Anyhow, it seems that this causes replication setup to fail; the TLS
doesn't get established because the CRL is not available to check the cert.

This does not occur in domainlevel=1 because in DL1, ipa-(replica|ca)-install sets up replication agreements itself, and configures a GSSAPI replica bind method.  In ipaserver/install/replication.py (note: isgssapi = True):

       if isgssapi:                     
           entry['nsds5replicatransportinfo'] = ['LDAP']       
           entry['nsds5replicabindmethod'] = ['SASL/GSSAPI']   
       else:                                                   
           entry['nsds5replicabinddn'] = [repl_man_dn]         
           entry['nsds5replicacredentials'] = [repl_man_passwd]
           entry['nsds5replicatransportinfo'] = ['TLS']        
           entry['nsds5replicabindmethod'] = ['simple']        

whereas in pkispawn (actually, ConfigurationUtils.java):

      if (replicationSecurity.equals("SSL")) {                             
          attrs.add(new LDAPAttribute("nsDS5ReplicaTransportInfo", "SSL"));
      } else if (replicationSecurity.equals("TLS")) {                      
          attrs.add(new LDAPAttribute("nsDS5ReplicaTransportInfo", "TLS"));
      }                                                                    

(so there is no option to set nsds5replicabindmethod=SASL/GSSAPI).

If my theories are correct (I'm yet to hack up the changes to test them), I
see three possible approaches to resolving (not mutually exclusive):

1. rollback the openldap change from RHEL-7.5.  not a permanent solution but it buys
   time to work out how to fix it properly.

2. update 389ds to make the CRLCHECK configurable.  If the default remains 'all'
   (or 'peer') then we'll also have to update IPA to tweak that setting to 'none'.

3. update Dogtag to allow setting GSS-API based replication agreements.  I'm not
   100% sure if this is feasible in DL0 or whether associated changes in IPA would
   also be required.

(In reply to Fraser Tweedale from comment #23)
> 2. update 389ds to make the CRLCHECK configurable.  If the default remains
> 'all'
>    (or 'peer') then we'll also have to update IPA to tweak that setting to
> 'none'.

I believe that this change makes sense (though I still would like to understand more about why the CRL is not available).  I have filed a bug against 389-ds-base to allow this to be configurable:

  https://bugzilla.redhat.com/show_bug.cgi?id=1541108

Nathan, yes I agree, we want to understand why the CRL processing isn't working.
Ideally we could make using CRLs or OCSP work, and even be the default for IPA.
But making 389 configurable in this regard means we can fix the regression quickly and then have to time to work out the rest of the puzzle.

Yes, it's caused by a change in openldap (building against OpenSSL instead of NSS now).  Openldap BZ is https://bugzilla.redhat.com/show_bug.cgi?id=1400578.

Kicking it back to ON_QA because there's a new 389 build (I think?) that should
resolve (part of?) the issues...

IPa-server: ipa-server-4.5.4-10.el7.x86_64

389-ds package version:
389-ds-base-libs-1.3.7.5-16.el7.x86_64
389-ds-base-1.3.7.5-16.el7.x86_64

Tested the bug with following observations:
1. Setup IPA master with KRA at domain Level 0.

RUNCMD: /usr/sbin/ipa-server-install --setup-dns --forwarder <forwarder_ip> --domain testrelm.test --realm TESTRELM.TEST --admin-password Secret123 --ds-password Secret123 -U --reverse-zone <reverse_zone_details> --allow-zone-overlap --setup-kra --domain-level=0

2. Prepare Replica file for Replica.
3. Setup Replica with kra against ipa-master at domain level0.
RUNCMD: /usr/sbin/ipa-replica-install -U --setup-dns --forwarder <forwarder_ip> --setup-ca --setup-kra --admin-password Secret123 --password Secret123 /var/lib/ipa/replica-info-replica.testrelm.test.gpg


Observations:
1. After step1, ipa-master is installed successfully
2. After step3, the replica installation is successful.

Thus on the basis of above observations, markign the status of bug to 'Verified'

*** Bug 1542839 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0925