Bug 888956

Summary: Cannot install an IPA Replica server with PKI-CA/Dogtag from a master with a large CRL
Product: Red Hat Enterprise Linux 6 Reporter: Dmitri Pal <dpal>
Component: ipaAssignee: Rob Crittenden <rcritten>
Status: CLOSED ERRATA QA Contact: Namita Soman <nsoman>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 6.4CC: mkosek, spoore, tlavigne
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ipa-3.0.0-20.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 09:31:32 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 895654    
Attachments:
Description Flags
Include ldif to set default maxbersize to larger value none

Description Dmitri Pal 2012-12-19 21:45:47 UTC 
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/freeipa/ticket/3314

There is a DIRSRV variable set during the PKI install: nsslapd-maxbersize.  This variable is -not- dynamically initialized.  Because of this, a restart is required for it to take effect.

When building a replica from a PKI-CA master with a large CRL, the installation fails during the replication phase : 

 Configuring certificate server: Estimated time 3 minutes 30 seconds
  [1/13]: creating certificate server user
  [2/13]: creating pki-ca instance
  [3/13]: configuring certificate server instance

/var/log/dirsrv/slapd-PKI-CA/error:

 [17/Dec/2012:15:50:22 -0800] - ERROR bulk import abandoned
 [17/Dec/2012:15:50:22 -0800] - import ipaca: Aborting all Import threads...
 [17/Dec/2012:15:50:33 -0800] - import ipaca: Import threads aborted.
 [17/Dec/2012:15:50:33 -0800] - import ipaca: Closing files...
 [17/Dec/2012:15:50:33 -0800] - import ipaca: Import failed.
 [17/Dec/2012:15:50:33 -0800] - process_bulk_import_op: NULL target sdn

/var/log/dirsrv/slapd-PKI-CA/access:

 [17/Dec/2012:15:50:22 -0800] conn=15 op=-1 fd=73 closed error 34 (Numerical result out of range) - B
Comment 1 Rob Crittenden 2013-01-08 15:35:35 UTC 
Created attachment 674915 [details]
Include ldif to set default maxbersize to larger value
Comment 2 Rob Crittenden 2013-01-08 15:58:01 UTC 
Fixed upstream.

master: cfe18944d643d8ec7c72a2e4c7081096d1fc2fb5

ipa-3-1: 56b756e382a26e257cd9d0dfe787514c800b2b93

ipa-3-0: 92ab3383492b73e99b84e499aeb21380959fbf6e
Comment 4 Scott Poore 2013-01-18 00:42:35 UTC 
Ok, so, I think I've almost got a test ready to verify this one.  Just need a little more info:

1.  How man CRL entries is "Large"?  Is 1 million enough?

2.  Is nsslapd-maxbersize in bytes?  

3.  Is nsslapd-maxbersize compared against MasterCRL?  So if I run the following, is the size of that file basically what mzxbersize limits?

sslget -v -n ipaCert -d /etc/httpd/alias -p $HPASS -r '/ca/ee/ca/getCRL?op=getCRL&crlIssuingPoint=MasterCRL' rhel6-4.testrelm.com:9444 > /tmp/MasterCRL.bin
Comment 5 Rob Crittenden 2013-01-18 14:14:48 UTC 
1. I'm not sure, I'll try to find out. On my system it looks like ~100 bytes per revoked entry (5 certs revoke, file size 580 bytes).

2. Yes

3. That is my understanding, yes.
Comment 6 Scott Poore 2013-01-28 20:34:15 UTC 
So if nsslapd-maxbersize is 209715200, and it's ~100bytes per, I'll need ~2million, not 1million in order to exceed the new limit.  So, to be safe here, I'll test with 3million.

Will that be enough to confirm the nsslapd-maxbersize limit?

Then I need to test ipa-replica-prepare/install?
Comment 7 Scott Poore 2013-01-30 17:27:45 UTC 
Ok, with Andrew, Rob, and Rich's help, I was able to reproduce this issue on RHEL6.4 by lowering the nsslapd-maxbersize on a replica before install and generating 200000 CRLs on the RHEL6.4 Master (which I did not change nsslapd-maxbersize on).

On Replica:

[root@rhel6-5 ~]# cat /usr/share/pki/ca/conf/database.ldif
dn: cn=config
changetype: modify
replace: nsslapd-maxbersize
nsslapd-maxbersize: 209715200

[root@rhel6-5 ~]# sed -i 's/nsslapd-maxbersize:.*$/nsslapd-maxbersize: 2097152/' /usr/share/pki/ca/conf/database.ldif

[root@rhel6-5 ~]# cat /usr/share/pki/ca/conf/database.ldif
dn: cn=config
changetype: modify
replace: nsslapd-maxbersize
nsslapd-maxbersize: 2097152


[root@rhel6-5 ~]# sftp root.122.64:/var/lib/ipa/replica-info-rhel6-5.testrelm.com.gpg /var/lib/ipa
Connecting to 192.168.122.64...
root.122.64's password: 
Fetching /var/lib/ipa/replica-info-rhel6-5.testrelm.com.gpg to /var/lib/ipa/replica-info-rhel6-5.testrelm.com.gpg
/var/lib/ipa/replica-info-rhel6-5.testrelm.com.gpg                   100%   35KB  35.1KB/s   00:00    

[root@rhel6-5 ~]# ipa-replica-install -U --setup-ca --setup-dns --forwarder=192.168.122.1 -w Secret123 -p Secret123 /var/lib/ipa/replica-info-rhel6-5.testrelm.com.gpg
Run connection check to master
Check connection from replica to remote master 'rhel6-4.testrelm.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK
   PKI-CA: Directory Service port (7389): OK

The following list of ports use UDP protocol and would need to be
checked manually:
   Kerberos KDC: UDP (88): SKIPPED
   Kerberos Kpasswd: UDP (464): SKIPPED

Connection from replica to master is OK.
Start listening on required ports for remote master check
Get credentials to log in to remote master
Execute check on remote master
Check connection from master to remote replica 'rhel6-5.testrelm.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK
   PKI-CA: Directory Service port (7389): OK

Connection from master to replica is OK.

Connection check OK
Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server for the CA (pkids): Estimated time 30 seconds
  [1/3]: creating directory server user
  [2/3]: creating directory server instance
  [3/3]: restarting directory server
Done configuring directory server for the CA (pkids).
Configuring certificate server (pki-cad): Estimated time 3 minutes 30 seconds
  [1/17]: creating certificate server user
  [2/17]: creating pki-ca instance
  [3/17]: configuring certificate server instance
ipa         : CRITICAL failed to configure ca instance Command '/usr/bin/perl /usr/bin/pkisilent ConfigureCA -cs_hostname rhel6-5.testrelm.com -cs_port 9445 -client_certdb_dir /tmp/tmp-7yyDWt -client_certdb_pwd XXXXXXXX -preop_pin qULwfhguKNnnXLElTWLB -domain_name IPA -admin_user admin -admin_email root@localhost -admin_password XXXXXXXX -agent_name ipa-ca-agent -agent_key_size 2048 -agent_key_type rsa -agent_cert_subject CN=ipa-ca-agent,O=TESTRELM.COM -ldap_host rhel6-5.testrelm.com -ldap_port 7389 -bind_dn cn=Directory Manager -bind_password XXXXXXXX -base_dn o=ipaca -db_name ipaca -key_size 2048 -key_type rsa -key_algorithm SHA256withRSA -save_p12 true -backup_pwd XXXXXXXX -subsystem_name pki-cad -token_name internal -ca_subsystem_cert_subject_name CN=CA Subsystem,O=TESTRELM.COM -ca_subsystem_cert_subject_name CN=CA Subsystem,O=TESTRELM.COM -ca_ocsp_cert_subject_name CN=OCSP Subsystem,O=TESTRELM.COM -ca_server_cert_subject_name CN=rhel6-5.testrelm.com,O=TESTRELM.COM -ca_audit_signing_cert_subject_name CN=CA Audit,O=TESTRELM.COM -ca_sign_cert_subject_name CN=Certificate Authority,O=TESTRELM.COM -external false -clone true -clone_p12_file ca.p12 -clone_p12_password XXXXXXXX -sd_hostname rhel6-4.testrelm.com -sd_admin_port 443 -sd_admin_name admin -sd_admin_password XXXXXXXX -clone_start_tls true -clone_uri https://rhel6-4.testrelm.com:443' returned non-zero exit status 255

Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

Configuration of CA failed

from /var/log/dirsrv/slapd-PKI-IPA/errors:

[30/Jan/2013:12:07:31 -0500] - ERROR bulk import abandoned
[30/Jan/2013:12:07:31 -0500] - import ipaca: Aborting all Import threads...
[30/Jan/2013:12:07:42 -0500] - import ipaca: Import threads aborted.
[30/Jan/2013:12:07:42 -0500] - import ipaca: Closing files...
[30/Jan/2013:12:07:42 -0500] - import ipaca: Import failed.
[30/Jan/2013:12:07:42 -0500] - process_bulk_import_op: NULL target sdn
Comment 8 Scott Poore 2013-01-30 17:55:08 UTC 
And, before officially verifying, here's the cleanup in between tests:

On Replica:

[root@rhel6-5 ~]# /usr/sbin/ipa-server-install --uninstall -U
Shutting down all IPA services
Removing IPA client configuration
Unconfiguring ntpd
Unconfiguring CA directory server

[root@rhel6-5 ~]# !pkiremove
pkiremove -pki_instance_root=/var/lib -pki_instance_name=pki-ca --force
PKI instance Deletion Utility ...

PKI instance Deletion Utility cleaning up instance ...

Contacting the security domain master to update the security domain

Removing selinux contexts

On Master:

[root@rhel6-4 ~]# ipa-replica-manage del rhel6-5.testrelm.com -f
Directory Manager password: 

'rhel6-4.testrelm.com' has no replication agreement for 'rhel6-5.testrelm.com'

[root@rhel6-4 ~]# kinit admin
Password for admin: 

[root@rhel6-4 ~]# ipa host-del rhel6-5.testrelm.com
ipa: ERROR: rhel6-5.testrelm.com: host not found
Comment 9 Scott Poore 2013-01-30 18:03:08 UTC 
Verified.

Version ::

ipa-server-3.0.0-24.el6.x86_64

Manual Test Results ::

Should be noted that this test is run with Master in same state as in comment #7 above with 200000 CRLs.

[root@rhel6-5 ~]# cat /usr/share/pki/ca/conf/database.ldif
dn: cn=config
changetype: modify
replace: nsslapd-maxbersize
nsslapd-maxbersize: 2097152

[root@rhel6-5 ~]# sed -i 's/nsslapd-maxbersize:.*$/nsslapd-maxbersize: 209715200/' /usr/share/pki/ca/conf/database.ldif

[root@rhel6-5 ~]# cat /usr/share/pki/ca/conf/database.ldifdn: cn=config
changetype: modify
replace: nsslapd-maxbersize
nsslapd-maxbersize: 209715200

[root@rhel6-5 ~]# sftp root.122.64:/var/lib/ipa/replica-info-rhel6-5.testrelm.com.gpg /var/lib/ipa
Connecting to 192.168.122.64...
root.122.64's password: 
Fetching /var/lib/ipa/replica-info-rhel6-5.testrelm.com.gpg to /var/lib/ipa/replica-info-rhel6-5.testrelm.com.gpg
/var/lib/ipa/replica-info-rhel6-5.testrelm.com.gpg                   100%   35KB  35.1KB/s   00:00    


[root@rhel6-5 ~]# ipa-replica-install -U --setup-ca --setup-dns --forwarder=192.168.122.1 -w Secret123 -p Secret123 /var/lib/ipa/replica-info-rhel6-5.testrelm.com.gpg
Run connection check to master
Check connection from replica to remote master 'rhel6-4.testrelm.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK
   PKI-CA: Directory Service port (7389): OK

The following list of ports use UDP protocol and would need to be
checked manually:
   Kerberos KDC: UDP (88): SKIPPED
   Kerberos Kpasswd: UDP (464): SKIPPED

Connection from replica to master is OK.
Start listening on required ports for remote master check
Get credentials to log in to remote master
Execute check on remote master
Check connection from master to remote replica 'rhel6-5.testrelm.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK
   PKI-CA: Directory Service port (7389): OK

Connection from master to replica is OK.

Connection check OK
Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server for the CA (pkids): Estimated time 30 seconds
  [1/3]: creating directory server user
  [2/3]: creating directory server instance
  [3/3]: restarting directory server
Done configuring directory server for the CA (pkids).
Configuring certificate server (pki-cad): Estimated time 3 minutes 30 seconds
  [1/17]: creating certificate server user
  [2/17]: creating pki-ca instance
  [3/17]: configuring certificate server instance
  [4/17]: disabling nonces
  [5/17]: creating RA agent certificate database
  [6/17]: importing CA chain to RA certificate database
  [7/17]: fixing RA database permissions
  [8/17]: setting up signing cert profile
  [9/17]: set up CRL publishing
  [10/17]: set certificate subject base
  [11/17]: enabling Subject Key Identifier
  [12/17]: setting audit signing renewal to 2 years
  [13/17]: configuring certificate server to start on boot
  [14/17]: configure certmonger for renewals
  [15/17]: configure clone certificate renewals
  [16/17]: configure Server-Cert certificate renewal
  [17/17]: Configure HTTP to proxy connections
Done configuring certificate server (pki-cad).
Restarting the directory and certificate servers
Configuring directory server (dirsrv): Estimated time 1 minute
  [1/31]: creating directory server user
  [2/31]: creating directory server instance
  [3/31]: adding default schema
  [4/31]: enabling memberof plugin
  [5/31]: enabling winsync plugin
  [6/31]: configuring replication version plugin
  [7/31]: enabling IPA enrollment plugin
  [8/31]: enabling ldapi
  [9/31]: disabling betxn plugins
  [10/31]: configuring uniqueness plugin
  [11/31]: configuring uuid plugin
  [12/31]: configuring modrdn plugin
  [13/31]: enabling entryUSN plugin
  [14/31]: configuring lockout plugin
  [15/31]: creating indices
  [16/31]: enabling referential integrity plugin
  [17/31]: configuring ssl for ds instance
  [18/31]: configuring certmap.conf
  [19/31]: configure autobind for root
  [20/31]: configure new location for managed entries
  [21/31]: restarting directory server
  [22/31]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress
Update in progress
Update in progress
Update in progress
Update succeeded
  [23/31]: adding replication acis
  [24/31]: setting Auto Member configuration
  [25/31]: enabling S4U2Proxy delegation
  [26/31]: initializing group membership
  [27/31]: adding master entry
  [28/31]: configuring Posix uid/gid generation
  [29/31]: enabling compatibility plugin
  [30/31]: tuning directory server
  [31/31]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc): Estimated time 30 seconds
  [1/9]: adding sasl mappings to the directory
  [2/9]: writing stash file from DS
  [3/9]: configuring KDC
  [4/9]: creating a keytab for the directory
  [5/9]: creating a keytab for the machine
  [6/9]: adding the password extension to the directory
  [7/9]: enable GSSAPI for replication
  [8/9]: starting the KDC
  [9/9]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring ipa_memcached
  [1/2]: starting ipa_memcached 
  [2/2]: configuring ipa_memcached to start on boot
Done configuring ipa_memcached.
Configuring the web interface (httpd): Estimated time 1 minute
  [1/12]: setting mod_nss port to 443
  [2/12]: setting mod_nss password file
  [3/12]: enabling mod_nss renegotiate
  [4/12]: adding URL rewriting rules
  [5/12]: configuring httpd
  [6/12]: setting up ssl
  [7/12]: publish CA cert
  [8/12]: creating a keytab for httpd
  [9/12]: clean up any existing httpd ccache
  [10/12]: configuring SELinux for httpd
  [11/12]: restarting httpd
  [12/12]: configuring httpd to start on boot
Done configuring the web interface (httpd).
Applying LDAP updates
Restarting the directory server
Restarting the KDC
Using reverse zone 122.168.192.in-addr.arpa.
Configuring DNS (named)
  [1/8]: adding NS record to the zone
  [2/8]: setting up reverse zone
  [3/8]: setting up our own record
  [4/8]: setting up kerberos principal
  [5/8]: setting up named.conf
  [6/8]: restarting named
  [7/8]: configuring named to start on boot
  [8/8]: changing resolv.conf to point to ourselves
Done configuring DNS (named).

Global DNS configuration in LDAP server is empty
You can use 'dnsconfig-mod' command to set global DNS options that
would override settings in local named.conf files

Restarting the web server
[root@rhel6-5 ~]#
Comment 11 errata-xmlrpc 2013-02-21 09:31:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0528.html