Bug 1235766 - Cannot add 4.1 Replica to 3.0 master Centos 6.6 -> Centos 7.1
Summary: Cannot add 4.1 Replica to 3.0 master Centos 6.6 -> Centos 7.1
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Fedora
Classification: Fedora
Component: freeipa
Version: 23
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: IPA Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-25 16:31 UTC by Yamakasi
Modified: 2016-01-26 16:07 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-01-26 16:07:19 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Yamakasi 2015-06-25 16:31:11 UTC
Description of problem:

When I install the 4.1 replica server (CentOS 7.1) I get this error on the new replica:

ipa         : CRITICAL CA DS schema check failed. Make sure the PKI
service on the remote master is operational.


When I restart IPA on the old 3.0 master (Centos 6.6) I get this:

    PKI-IPA...[22/Jun/2015:22:50:36 +0200] attr_syntax_create - Error:
the EQUALITY matching rule [caseIgnoreIA5Match] is not compatible with
the syntax [1.3.6.1.4.1.1466.115.121.1.15] for the attribute [dc]
[22/Jun/2015:22:50:36 +0200] attr_syntax_create - Error: the SUBSTR
matching rule [caseIgnoreIA5SubstringsMatch] is not compatible with
the syntax [1.3.6.1.4.1.1466.115.121.1.15] for the attribute [dc]
                                                           [  OK  ]


Version-Release number of selected component (if applicable):

3.0 master
4.1 replica

How reproducible:

Follow these steps for upgrading:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/upgrading.html

Actual results:

Messes up Master Schemas and cannot add the replica. Master still works OK

Expected results:

Have a 4.1 replica of the 3.0 master

Additional info:

Comment 1 Petr Vobornik 2015-06-26 07:32:15 UTC
Hello Yamakasi,

the error you see in the ds could be a false positive. 

I want to make sure that you run steps 3. and 4. of https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/migrating-ipa-proc.html

If yes and you still receive the error please attach:
* /var/log/ipareplica/ipareplica-install.log
* /var/log

Comment 2 Petr Vobornik 2015-06-26 07:39:30 UTC
Comment 1 was sent prematurely:

the second log should be:
/var/log/dirsrv/slapd-YOUR-REALM/errors  (of both servers)

Besides that, based on your description, this bug is a most likely a duplicate of bug 1167196 , there is also related bug 1224769.

Comment 3 Yamakasi 2015-06-27 18:23:17 UTC
I checked the copy-schema-to-ca.py and it's the right version of what it should be.

I don't see any strange errors on the master, but the replica gives me the following:


Check SSH connection to remote master
Execute check on remote master
Check connection from master to remote replica 'ipa-01-71.domain.local':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

Connection from master to replica is OK.

Connection check OK
Using reverse zone(s) 3.16.172.in-addr.arpa.
ipa         : CRITICAL CA DS schema check failed. Make sure the PKI service on the remote master is operational.

Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

Invalid password
[root@ipa-01-71 ~]# tail -f /var/log/ipareplica-install.log
  File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 469, in result3
    resp_ctrl_classes=resp_ctrl_classes

  File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 476, in result4
    ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)

  File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99, in _ldap_call
    result = func(*args,**kwargs)

2015-06-27T18:20:51Z DEBUG The ipa-replica-install command failed, exception: INVALID_CREDENTIALS: {'desc': 'Invalid credentials'}


But the credentials are OK.

Comment 4 Petr Vobornik 2015-06-29 08:35:28 UTC
We need more from the log, this snippet doesn't even contain the full traceback.

But the error message indeed indicates that the supplied Directory Manager's password is incorrect. 

Are you able to do following LDAP search with the password on the 3.0 master:

 ldapsearch  -D "cn=directory manager" -W -s ONE

Comment 5 Yamakasi 2015-06-29 11:31:10 UTC
I don't see any more logging in the logs, I need to recheck but I'm pretty sure there isn't more.

I can do an ldap search, that is such strange.

Comment 6 Petr Vobornik 2015-06-29 14:03:08 UTC
tail with `-f` options prints couple of last lines of the log and then the lines which are appended to the log file while the command is running.   

I bet that `less /var/log/ipareplica-install.log` will show you much more.

Comment 7 Yamakasi 2015-07-03 12:20:31 UTC
HI Petr,

There are more lines, only debug ones and no errors.

It really fails on this Critical error.


2015-06-29T11:28:52Z DEBUG /usr/sbin/ipa-replica-install was invoked with argument "/var/lib/ipa/replica-info-ipa-01-71.domain.local.gpg" and options: {'no_forwarders': True, 'conf_ssh': True, 'skip_schema_check': False, 'ui_redirect': True, 'trust_sshfp': False, 'unattended': True, 'ip_addresses': [CheckedIPAddress('172.16.3.151')], 'no_host_dns': False, 'mkhomedir': False, 'no_reverse': False, 'setup_dns': True, 'create_sshfp': True, 'conf_sshd': True, 'forwarders': None, 'debug': False, 'conf_ntp': False, 'setup_ca': True, 'skip_conncheck': False, 'reverse_zones': []}
2015-06-29T11:28:52Z DEBUG IPA version 4.1.0-18.el7.centos.3
2015-06-29T11:28:52Z DEBUG Loading Index file from '/var/lib/ipa-client/sysrestore/sysrestore.index'
2015-06-29T11:28:52Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2015-06-29T11:28:52Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2015-06-29T11:28:52Z DEBUG Starting external process
2015-06-29T11:28:52Z DEBUG args='/usr/sbin/httpd' '-t' '-D' 'DUMP_VHOSTS'
2015-06-29T11:28:52Z DEBUG Process finished, return code=0
2015-06-29T11:28:52Z DEBUG stdout=VirtualHost configuration:
*:8443                 is a NameVirtualHost
         default server ipa-01-71.domain.local (/etc/httpd/conf.d/nss.conf:86)
         port 8443 namevhost ipa-01-71.domain.local (/etc/httpd/conf.d/nss.conf:86)
         port 8443 namevhost ipa-01-71.domain.local (/etc/httpd/conf.d/nss.conf:86)

2015-06-29T11:28:52Z DEBUG stderr=
2015-06-29T11:29:05Z DEBUG Starting external process
2015-06-29T11:29:05Z DEBUG args='/usr/bin/gpg-agent' '--batch' '--homedir' '/tmp/tmpmbCjioipa/ipa-7m8PK5/.gnupg' '--daemon' '/usr/bin/gpg' '--batch' '--homedir' '/tmp/tmpmbCjioipa/ipa-7m8PK5/.gnupg' '--passphrase-fd' '0' '--yes' '--no-tty' '-o' '/tmp/tmpmbCjioipa/files.tar' '-d' '/var/lib/ipa/replica-info-ipa-01-71.domain.local.gpg'
2015-06-29T11:29:05Z DEBUG Process finished, return code=0
2015-06-29T11:29:05Z DEBUG Starting external process
2015-06-29T11:29:05Z DEBUG args='tar' 'xf' '/tmp/tmpmbCjioipa/files.tar' '-C' '/tmp/tmpmbCjioipa'
2015-06-29T11:29:05Z DEBUG Process finished, return code=0
2015-06-29T11:29:05Z DEBUG stdout=
2015-06-29T11:29:05Z DEBUG stderr=
2015-06-29T11:29:05Z DEBUG Installing replica file with version 300 (0 means no version in prepared file).
2015-06-29T11:29:05Z DEBUG Check if ipa-01-71.domain.local is a primary hostname for localhost
2015-06-29T11:29:05Z DEBUG Primary hostname for localhost: ipa-01-71.domain.local
2015-06-29T11:29:05Z DEBUG Search DNS for ipa-01-71.domain.local
2015-06-29T11:29:05Z DEBUG Check if ipa-01-71.domain.local is not a CNAME
2015-06-29T11:29:05Z DEBUG Check reverse address of 172.16.3.151
2015-06-29T11:29:05Z DEBUG Found reverse name: ipa-01-71.domain.local
2015-06-29T11:29:05Z DEBUG Check if ipa-01.domain.local is a primary hostname for localhost
2015-06-29T11:29:05Z DEBUG Primary hostname for localhost: ipa-01.domain.local
2015-06-29T11:29:05Z DEBUG Search DNS for ipa-01.domain.local
2015-06-29T11:29:05Z DEBUG Check if ipa-01.domain.local is not a CNAME
2015-06-29T11:29:05Z DEBUG Check reverse address of 172.16.3.251
2015-06-29T11:29:05Z DEBUG Found reverse name: ipa-01.domain.local
2015-06-29T11:29:05Z DEBUG Starting external process
2015-06-29T11:29:05Z DEBUG args='/usr/sbin/ipa-replica-conncheck' '--master' 'ipa-01.domain.local' '--auto-master-check' '--realm' 'domain.local' '--principal' 'admin' '--hostname' 'ipa-01-71.domain.local' '--check-ca'
2015-06-29T11:29:15Z DEBUG Process finished, return code=0
2015-06-29T11:29:15Z DEBUG Starting external process
2015-06-29T11:29:15Z DEBUG args='/sbin/ip' '-family' 'inet' '-oneline' 'address' 'show'
2015-06-29T11:29:15Z DEBUG Process finished, return code=0
2015-06-29T11:29:15Z DEBUG stdout=1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: eth0    inet 172.16.3.151/24 brd 172.16.3.255 scope global dynamic eth0\       valid_lft 489sec preferred_lft 489sec

2015-06-29T11:29:15Z DEBUG stderr=
2015-06-29T11:29:15Z DEBUG importing all plugin modules in '/usr/lib/python2.7/site-packages/ipalib/plugins'...
2015-06-29T11:29:15Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/aci.py'
2015-06-29T11:29:15Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/automember.py'
2015-06-29T11:29:15Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/automount.py'
2015-06-29T11:29:15Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/baseldap.py'
2015-06-29T11:29:15Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/batch.py'
2015-06-29T11:29:15Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/cert.py'
2015-06-29T11:29:15Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/config.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/delegation.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/dns.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/group.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/hbacrule.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/hbacsvc.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/hbacsvcgroup.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/hbactest.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/host.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/hostgroup.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/idrange.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/idviews.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/internal.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/kerberos.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/krbtpolicy.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/migration.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/misc.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/netgroup.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/otptoken.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/otptoken_yubikey.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/passwd.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/permission.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/ping.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/pkinit.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/privilege.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/pwpolicy.py'
2015-06-29T11:29:16Z DEBUG Starting external process
2015-06-29T11:29:16Z DEBUG args='klist' '-V'
2015-06-29T11:29:16Z DEBUG Process finished, return code=0
2015-06-29T11:29:16Z DEBUG stdout=Kerberos 5 version 1.12.2

2015-06-29T11:29:16Z DEBUG stderr=
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/radiusproxy.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/realmdomains.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/role.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/rpcclient.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/selfservice.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/selinuxusermap.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/service.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/sudocmd.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/sudocmdgroup.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/sudorule.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/trust.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/user.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipalib/plugins/virtual.py'
2015-06-29T11:29:16Z DEBUG importing all plugin modules in '/usr/lib/python2.7/site-packages/ipaserver/install/plugins'...
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/adtrust.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/baseupdate.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/ca_renewal_master.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/dns.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/fix_replica_agreements.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/rename_managed.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/update_idranges.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/update_managed_permissions.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/update_pacs.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/update_passsync.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/update_referint.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/update_services.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/update_uniqueness.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/updateclient.py'
2015-06-29T11:29:16Z DEBUG importing plugin module '/usr/lib/python2.7/site-packages/ipaserver/install/plugins/upload_cacrt.py'
2015-06-29T11:29:17Z DEBUG group dirsrv exists
2015-06-29T11:29:17Z DEBUG user dirsrv exists
2015-06-29T11:29:17Z DEBUG Created connection context.ldap2_40587920
2015-06-29T11:29:17Z DEBUG flushing ldaps://ipa-01.domain.local from SchemaCache
2015-06-29T11:29:17Z DEBUG retrieving schema for SchemaCache url=ldaps://ipa-01.domain.local conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x3c6b6c8>
2015-06-29T11:29:18Z DEBUG Created connection context.ldap2
2015-06-29T11:29:18Z DEBUG flushing ldaps://ipa-01.domain.local from SchemaCache
2015-06-29T11:29:18Z DEBUG retrieving schema for SchemaCache url=ldaps://ipa-01.domain.local conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x429d488>
2015-06-29T11:29:18Z DEBUG Destroyed connection context.ldap2
2015-06-29T11:29:18Z DEBUG Check forward/reverse DNS resolution
2015-06-29T11:29:18Z DEBUG Search DNS server ipa-01.domain.local (['172.16.3.251', '172.16.3.251', '172.16.3.251']) for ipa-01.domain.local
2015-06-29T11:29:18Z DEBUG Check reverse address 172.16.3.251 (ipa-01.domain.local)
2015-06-29T11:29:18Z DEBUG Address 172.16.3.251 resolves to: ipa-01.domain.local.. 
2015-06-29T11:29:18Z DEBUG Search DNS server ipa-01.domain.local (['172.16.3.251', '172.16.3.251', '172.16.3.251']) for ipa-01-71.domain.local
2015-06-29T11:29:18Z DEBUG Check reverse address 172.16.3.151 (ipa-01-71.domain.local)
2015-06-29T11:29:18Z DEBUG Address 172.16.3.151 resolves to: ipa-01-71.domain.local.. 
2015-06-29T11:29:18Z DEBUG Destroyed connection context.ldap2_40587920
2015-06-29T11:29:18Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2015-06-29T11:29:18Z DEBUG Checking if IPA schema is present in ldap://ipa-01.domain.local:7389
2015-06-29T11:29:18Z CRITICAL CA DS schema check failed. Make sure the PKI service on the remote master is operational.
2015-06-29T11:29:18Z DEBUG   File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 646, in run_script
    return_value = main_function()

  File "/usr/sbin/ipa-replica-install", line 691, in main
    cainstance.replica_ca_install_check(config)

  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1798, in replica_ca_install_check
    config.dirman_password)

  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 663, in simple_bind_s
    return self.conn.simple_bind_s(who, cred, serverctrls, clientctrls)

  File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 208, in simple_bind_s
    resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all=1,timeout=self.timeout)

  File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 469, in result3
    resp_ctrl_classes=resp_ctrl_classes

  File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 476, in result4
    ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)

  File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99, in _ldap_call
    result = func(*args,**kwargs)

2015-06-29T11:29:18Z DEBUG The ipa-replica-install command failed, exception: INVALID_CREDENTIALS: {'desc': 'Invalid credentials'}

Comment 8 Petr Vobornik 2015-07-09 13:09:07 UTC
Weird, It still points to wrong credentials.

Could you execute following search from 4.1 replica:
ldapsearch -H ldap://ipa-01.domain
.local:7389  -D "cn=directory manager" -W -s ONE -b o=ipaca

Does it return successfully with e.g. "numEntries: 9" on the end of response

Comment 9 Jan Kurik 2015-07-15 13:54:59 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23

Comment 10 Petr Vobornik 2015-07-16 16:58:43 UTC
waiting for answer for question in comment 8

Comment 11 Yamakasi 2015-07-20 08:36:59 UTC
Hi Petr,

Sorry I was on something else that had priority.

I cannot run that command on the replica as the installation failed.

I even tried it with resetting the Directory Manager password to something very simple but that didn't change it also.

Same PKI error.

Comment 12 Petr Vobornik 2015-07-20 09:03:30 UTC
you don't have to have ipa-server installed on the server in order to run ldapsearch against other server.

When changing Directory Manager password, make sure to follow: 

http://www.freeipa.org/page/Howto/Change_Directory_Manager_Password

Comment 13 Yamakasi 2015-07-20 09:09:35 UTC
True, but is there something wrong in that command as I wasn't able to run it on the ipa master itself also.

ldapsearch  -D "cn=directory manager" -W -s ONE

gives me reply, your command gives me indeed "ldap_bind: Invalid credentials (49)"

Comment 14 Petr Vobornik 2015-07-20 11:37:13 UTC
For the record, I've just discussed another occurrence of this bug and the cause was that REALM dirsrv instance had a different directory manager password than the PKI(CA) instance. 

Could be checked by comparing nsslapd-rootpw in 
- /etc/dirsrv/slapd-MY-REALM-TEST/dse.ldif
- /etc/dirsrv/slapd-PKI-IPA/dse.ldif

Comment 15 Yamakasi 2015-07-20 16:55:34 UTC
OK, the password issue is solved. This was an issue between the CA and LDAP password.

Now the install starts but ends with:

2015-07-20T11:08:44Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2015-07-20T11:08:44Z DEBUG Starting external process
2015-07-20T11:08:44Z DEBUG args='/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmp3IqrOU'
2015-07-20T11:12:36Z DEBUG Process finished, return code=1
2015-07-20T11:12:36Z DEBUG stdout=Loading deployment configuration from /tmp/tmp3IqrOU.
Installing CA into /var/lib/pki/pki-tomcat.
Storing deployment configuration into /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg.

Installation failed.


2015-07-20T11:12:36Z DEBUG stderr=pkispawn    : WARNING  ....... unable to validate security domain user/password through REST interface.
Interface not available
pkispawn    : ERROR    ....... Exception from Java Configuration Servlet: Error while updating security domain:
java.io.IOException: java.io.IOException: SocketException cannot read on socket

2015-07-20T11:12:36Z CRITICAL failed to configure ca instance Command ''/usr/sbin/pkispawn' '-s' 'CA' '-f' '/tmp/tmp3IqrOU'' returned non-zero exit status 1
2015-07-20T11:12:36Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 382, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 372, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 673, in __spawn_instance
    raise RuntimeError('Configuration of CA failed')
RuntimeError: Configuration of CA failed

2015-07-20T11:12:36Z DEBUG   [error] RuntimeError: Configuration of CA failed
2015-07-20T11:12:36Z DEBUG   File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 646, in run_script
    return_value = main_function()

  File "/usr/sbin/ipa-replica-install", line 703, in main
    CA = cainstance.install_replica_ca(config)

  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1869, in install_replica_ca
    subject_base=config.subject_base)

  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 520, in configure_instance
    self.start_creation(runtime=210)

  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 382, in start_creation
    run_step(full_msg, method)

  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 372, in run_step
    method()

  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 673, in __spawn_instance
    raise RuntimeError('Configuration of CA failed')

2015-07-20T11:12:36Z DEBUG The ipa-replica-install command failed, exception: RuntimeError: Configuration of CA failed


AND:


2015-07-20 11:08:48 pkispawn    : DEBUG    ........... chown 995:993 /root/.dogtag/pki-tomcat/ca/pkcs12_password.conf
2015-07-20 11:08:48 pkispawn    : INFO     ....... executing 'certutil -N -d /tmp/tmp-rbfc7m -f /root/.dogtag/pki-tomcat/ca/password.conf'
2015-07-20 11:08:48 pkispawn    : INFO     ....... executing 'systemctl daemon-reload'
2015-07-20 11:08:48 pkispawn    : INFO     ....... executing 'systemctl start pki-tomcatd'
2015-07-20 11:08:49 pkispawn    : DEBUG    ........... No connection - server may still be down
2015-07-20 11:08:49 pkispawn    : DEBUG    ........... No connection - exception thrown: [Errno 111] Connection refused
2015-07-20 11:08:50 pkispawn    : DEBUG    ........... No connection - server may still be down
2015-07-20 11:08:50 pkispawn    : DEBUG    ........... No connection - exception thrown: [Errno 111] Connection refused
2015-07-20 11:08:51 pkispawn    : DEBUG    ........... No connection - server may still be down
2015-07-20 11:08:51 pkispawn    : DEBUG    ........... No connection - exception thrown: [Errno 111] Connection refused
2015-07-20 11:08:52 pkispawn    : DEBUG    ........... No connection - server may still be down
2015-07-20 11:08:52 pkispawn    : DEBUG    ........... No connection - exception thrown: [Errno 111] Connection refused
2015-07-20 11:08:53 pkispawn    : DEBUG    ........... No connection - server may still be down
2015-07-20 11:08:53 pkispawn    : DEBUG    ........... No connection - exception thrown: [Errno 111] Connection refused
2015-07-20 11:08:54 pkispawn    : DEBUG    ........... No connection - server may still be down
2015-07-20 11:08:54 pkispawn    : DEBUG    ........... No connection - exception thrown: [Errno 111] Connection refused
2015-07-20 11:09:06 pkispawn    : DEBUG    ........... <?xml version="1.0" encoding="UTF-8" standalone="no"?><XMLResponse><State>0</State><Type>CA</Type><St$
2015-07-20 11:09:07 pkispawn    : INFO     ....... constructing PKI configuration data.
2015-07-20 11:09:07 pkispawn    : INFO     ....... configuring PKI configuration data.
2015-07-20 11:12:35 pkispawn    : ERROR    ....... Exception from Java Configuration Servlet: Error while updating security domain: java.io.IOException: jav$
2015-07-20 11:12:35 pkispawn    : DEBUG    ....... Error Type: HTTPError
2015-07-20 11:12:35 pkispawn    : DEBUG    ....... Error Message: 500 Server Error: Internal Server Error
2015-07-20 11:12:35 pkispawn    : DEBUG    .......   File "/usr/sbin/pkispawn", line 463, in main
    rv = instance.spawn(deployer)
  File "/usr/lib/python2.7/site-packages/pki/server/deployment/scriptlets/configuration.py", line 126, in spawn
    json.dumps(data, cls=pki.encoder.CustomTypeEncoder))
  File "/usr/lib/python2.7/site-packages/pki/server/deployment/pkihelper.py", line 3211, in configure_pki_data
    response = client.configure(data)
  File "/usr/lib/python2.7/site-packages/pki/system.py", line 80, in configure
    r = self.connection.post('/rest/installer/configure', data, headers)
  File "/usr/lib/python2.7/site-packages/pki/client.py", line 64, in post
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 638, in raise_for_status
    raise http_error



Might it be that I need to update the cacert.p12 on the master where I generate the replica file ?

Comment 16 Yamakasi 2015-07-21 13:52:27 UTC
Created attachment 1054405 [details]
pki-tomcat-ca-debug loggin after replica install from replica

Comment 17 Yamakasi 2015-07-21 14:04:42 UTC
Created attachment 1054409 [details]
pki-tomcat-ca-debug loggin after replica install from replica

Comment 18 Yamakasi 2015-07-22 12:33:52 UTC
It seems that when I reset the ldap password something goes wrong.

- An ldapsearch can be done with the NEW set password 

- An ldappasswd can only be done with the OLD password

Dogtag not updated in some way ?

Comment 19 Yamakasi 2015-07-22 20:46:30 UTC
All passwords reset and double checked, it goes wrong here after getting the XML from the master. 

It can receive the XML:

2015-07-22 22:29:21 pkispawn    : INFO     ....... modifying '/root/.dogtag/pki-tomcat/ca/password.conf'
2015-07-22 22:29:21 pkispawn    : DEBUG    ........... chmod 660 /root/.dogtag/pki-tomcat/ca/password.conf
2015-07-22 22:29:21 pkispawn    : DEBUG    ........... chown 0:0 /root/.dogtag/pki-tomcat/ca/password.conf
2015-07-22 22:29:21 pkispawn    : INFO     ....... generating '/root/.dogtag/pki-tomcat/ca/pkcs12_password.conf'
2015-07-22 22:29:21 pkispawn    : INFO     ....... modifying '/root/.dogtag/pki-tomcat/ca/pkcs12_password.conf'
2015-07-22 22:29:21 pkispawn    : DEBUG    ........... chmod 660 /root/.dogtag/pki-tomcat/ca/pkcs12_password.conf
2015-07-22 22:29:21 pkispawn    : DEBUG    ........... chown 995:993 /root/.dogtag/pki-tomcat/ca/pkcs12_password.conf
2015-07-22 22:29:21 pkispawn    : INFO     ....... executing 'certutil -N -d /tmp/tmp-NYQvku -f /root/.dogtag/pki-tomcat/ca/password.conf'
2015-07-22 22:29:21 pkispawn    : INFO     ....... executing 'systemctl daemon-reload'
2015-07-22 22:29:21 pkispawn    : INFO     ....... executing 'systemctl start pki-tomcatd'
2015-07-22 22:29:22 pkispawn    : DEBUG    ........... No connection - server may still be down
2015-07-22 22:29:22 pkispawn    : DEBUG    ........... No connection - exception thrown: [Errno 111] Connection refused
2015-07-22 22:29:23 pkispawn    : DEBUG    ........... No connection - server may still be down
2015-07-22 22:29:23 pkispawn    : DEBUG    ........... No connection - exception thrown: [Errno 111] Connection refused
2015-07-22 22:29:24 pkispawn    : DEBUG    ........... No connection - server may still be down
2015-07-22 22:29:24 pkispawn    : DEBUG    ........... No connection - exception thrown: [Errno 111] Connection refused
2015-07-22 22:29:37 pkispawn    : DEBUG    ........... <?xml version="1.0" encoding="UTF-8" standalone="no"?><XMLResponse><State>0
</State><Type>CA</Type><Status>running</Status><Version>10.1.2-7.el7</Version></XMLResponse>
2015-07-22 22:29:38 pkispawn    : INFO     ....... constructing PKI configuration data.
2015-07-22 22:29:38 pkispawn    : INFO     ....... configuring PKI configuration data.
2015-07-22 22:29:40 pkispawn    : ERROR    ....... Exception from Java Configuration Servlet: Failed to obtain installation token from security domain
2015-07-22 22:29:40 pkispawn    : DEBUG    ....... Error Type: HTTPError
2015-07-22 22:29:40 pkispawn    : DEBUG    ....... Error Message: 500 Server Error: Internal Server Error
2015-07-22 22:29:40 pkispawn    : DEBUG    .......   File "/usr/sbin/pkispawn", line 463, in main
    rv = instance.spawn(deployer)
  File "/usr/lib/python2.7/site-packages/pki/server/deployment/scriptlets/configuration.py", line 126, in spawn
    json.dumps(data, cls=pki.encoder.CustomTypeEncoder))
  File "/usr/lib/python2.7/site-packages/pki/server/deployment/pkihelper.py", line 3211, in configure_pki_data
    response = client.configure(data)
  File "/usr/lib/python2.7/site-packages/pki/system.py", line 80, in configure
    r = self.connection.post('/rest/installer/configure', data, headers)
  File "/usr/lib/python2.7/site-packages/pki/client.py", line 64, in post
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 638, in raise_for_status
    raise http_error

Comment 20 Yamakasi 2015-07-22 21:58:51 UTC
After some digging and some other bug I found, I think it's the XML which is not OK

23:50 < YamakasY> ok, maybe some xml issue on the format ?
23:52 < YamakasY> kewl found it !
23:52 < YamakasY> securityDomainName=null, securityDomainUser=admin, securityDomainPassword=XXXX
23:53 < YamakasY> and that is indeed NOT in the XML

<XMLResponse><DomainInfo><?xml version="1.0" encoding="UTF-8" standalone="no"?><DomainInfo><Name>IPA</Name><CAList><CA><Host>ipa-01.domain.local</Host><SecurePort>443</SecurePort><SecureAgentPort>443</SecureAgentPort><SecureAdminPort>443</SecureAdminPort><SecureEEClientAuthPort>443</SecureEEClientAuthPort><UnSecurePort>80</UnSecurePort><Clone>FALSE</Clone><SubsystemName>pki-cad</SubsystemName><DomainManager>TRUE</DomainManager></CA><SubsystemCount>1</SubsystemCount></CAList><OCSPList><SubsystemCount>0</SubsystemCount></OCSPList><KRAList><SubsystemCount>0</SubsystemCount></KRAList><RAList><SubsystemCount>0</SubsystemCount></RAList><TKSList><SubsystemCount>0</SubsystemCount></TKSList><TPSList><SubsystemCount>0</SubsystemCount></TPSList></DomainInfo></DomainInfo><Status>0</Status></XMLResponse>


Refering to this:

https://fedorahosted.org/pki/ticket/1235

Where I came across because of some 3-topic mailinglist item which had the same issue only on 3.3


I don't see a REALM or Domain ?

Comment 21 Yamakasi 2015-07-24 12:04:56 UTC
OK, password was still not updated well using the docs, this is covered now with help from Ade.

Now the replica fails on:


[23/Jul/2015:18:13:11][http-bio-8443-exec-3]: Cloning a domain master
[23/Jul/2015:18:13:11][http-bio-8443-exec-3]: WizardPanelBase updateDomainXML start hostname=ipa-01.domain.local port=443
[23/Jul/2015:18:13:11][http-bio-8443-exec-3]: updateSecurityDomain: failed to update security domain using admin port 443:org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 50; White spaces are required between publicId and systemId.
[23/Jul/2015:18:13:11][http-bio-8443-exec-3]: updateSecurityDomain: now trying agent port with client auth
[23/Jul/2015:18:13:11][http-bio-8443-exec-3]: WizardPanelBase updateDomainXML start hostname=ipa-01.domain.local port=443
[23/Jul/2015:18:13:11][http-bio-8443-exec-3]: updateDomainXML() nickname=subsystemCert cert-pki-ca
[23/Jul/2015:18:13:12][http-bio-8443-exec-3]: WizardPanelBase updateDomainXML: status=1

Comment 22 Ade Lee 2015-07-24 13:51:48 UTC
The replica is almost complete, but I meed a few things to diagnose where it went wrong.  Specifically ..

debug log on the master /var/log/pki-ca/debug
catalina.out on the master: /var/log/pki-ca/catalina,out
debug log pn replica /var/log/pki/pki-tomcat/ca/debug
journal on the replica : journalctl -u pki-tomcatd

IPA replica instal log -- so I can see the exception thrown.

Ade

Comment 23 Yamakasi 2015-07-26 01:09:14 UTC
OK, as we know I need to update the certificate but I get an error on this ldapmodify:

dn: uid=CA-ipa.example.com-9443,ou=people,o=ipaca
changetype: modify
replace: usercertificate
usercertificate:< file:///tmp/subsystem.der
-
replace: description
description: 2;123;CN=Certificate Authority,O=EXAMPLE.COM;CN=CA Subsystem,O=EXAMPLE.COM

ldapmodify: wrong attributeType at line 4, entry "uid=CA-ipa.example.com-9443,ou=people,o=ipaca"

Comment 24 Petr Vobornik 2015-11-18 13:47:51 UTC
What is the result of this bug? Can it be closed?

If others experience it: if the 6.7 was a master with externally signed CA cert, there is a bug 1256039 which says that there the cause is missing PKI subsystem user.

Workaround text from the bug:

"""
3. Based on master's system log it looks like the subsystem certificate provided by the replica cannot be mapped into a user on the master:

1850.TP-Processor8 - [23/Oct/2015:12:47:30 CDT] [6] [3] Cannot authenticate agent with certificate Serial 0x3 Subject DN CN=CA Subsystem,O=EXAMPLE.COM. Error: User not found
1850.TP-Processor8 - [23/Oct/2015:12:47:30 CDT] [3] [3] Servlet caUpdateDomainXML: Failed to authorize: Invalid Credential..

To help troubleshooting ideally this message should be logged in the debug log as well in the next RHEL 6 update.

4. The authentication problem might be caused by a missing subsystem user (bug #1225589) and there's already a tool to restore it. However, before running the script, please run this command on the master to verify the problem:

$ pki-server ca-group-member-find "Subsystem Group"

Ideally it should return a user ID "CA-<hostname>-9443" and the description attribute should contain the subsystem certificate in this format "<version>;<serial>;<issuer DN>;<subject DN>".

If that's not the case, please run this tool to restore the subsystem user:

$ python /usr/share/pki/scripts/restore-subsystem-user.py

Then run this command again to verify the fix:

$ pki-server ca-group-member-find "Subsystem Group"

If everything works well, please try installing the replica again.
"""

Comment 25 Petr Vobornik 2016-01-26 16:07:19 UTC
RHEL 6.7 recently received update which fixes some issues with migration procedure from 3.0 to 4.x. See the bugs in comment 24.

Closing this bug due to lack of activity.


Note You need to log in before you can comment on or make changes to this bug.