Bug 894131

Summary: ipa-replica-install fails to add idnssoaserial for a new zone
Product: Red Hat Enterprise Linux 6 Reporter: Scott Poore <spoore>
Component: ipaAssignee: Rob Crittenden <rcritten>
Status: CLOSED ERRATA QA Contact: Namita Soman <nsoman>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 6.4CC: mkosek, pspacek, tlavigne
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ipa-3.0.0-22.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 895083 (view as bug list) Environment:
Last Closed: 2013-02-21 09:32:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 895083    
Bug Blocks: 902691    

Description Scott Poore 2013-01-10 20:03:38 UTC 
Description of problem:

It looks like ipa-replica-install doesn't always properly add idnssoaserial for new entries.  From testing, I'm seeing a zone get added but it's missing that data.  At least that's not getting replicate back to the master.  

In order to test in my isolated environment, I have to delete the existing reverse zone because the master and replica are on same virtual network.  And in test scripts, I can't currently guarantee servers will be on different networks, so that does the same.

After initial Master install, I see this:

[root@rhel6-1 shared]# ipa dnszone-show 122.168.192.in-addr.arpa.
  Zone name: 122.168.192.in-addr.arpa.
  Authoritative nameserver: rhel6-1.testrelm.com.
  Administrator e-mail address: hostmaster.testrelm.com.
  SOA serial: 1357837632
  SOA refresh: 3600
  SOA retry: 900
  SOA expire: 1209600
  SOA minimum: 3600
  Active zone: TRUE
  Allow query: any;
  Allow transfer: none;

Afterward ipa dnszone-del and ipa-replica-install, I see this:

[root@rhel6-1 log]# ipa dnszone-find
  Zone name: 122.168.192.in-addr.arpa.
  Authoritative nameserver: rhel6-2.testrelm.com.
  Administrator e-mail address: hostmaster.testrelm.com.
  SOA refresh: 3600
  SOA retry: 900
  SOA expire: 1209600
  SOA minimum: 3600
  Active zone: TRUE
  Allow query: any;
  Allow transfer: none;


No "SOA serial" option there.  Then, if I try to re-run ipa-replica-prepare, that's when I see errors that led me here.


Version-Release number of selected component (if applicable):


How reproducible:
always (at least with ipa-replica-install options listed):

Steps to Reproduce:

On Master:
1. setup IPA Master server
2. ipa dnszone-del <reverse zone for replica if it exists>
3. ipa-replica-prepare (with no --ip-address option)

On Replica:
4. sftp replica info gpg file
5. ipa-replica-install -U --setup-dns --no-forwarders -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$s_short.$DOMAIN.gpg
  
Actual results:
ipa dnszone-show <reverse zone for replica>
does not show SOA serial value.

However, looks like it was created with one:

log shows that it should have been added with the the idnssoaserial value though:

2013-01-10T18:42:51Z DEBUG   [2/8]: setting up reverse zone
2013-01-10T18:42:51Z DEBUG raw: dnszone_add(u'122.168.192.in-addr.arpa.', idnssoamname=u'rhel6-2.testre
lm.com.', idnssoarname=u'hostmaster.testrelm.com', idnsupdatepolicy=u'grant TESTRELM.COM krb5-subdomain
 122.168.192.in-addr.arpa. PTR;', idnsallowdynupdate=True, idnsallowquery=u'any', idnsallowtransfer=u'n
one', force=True, ip_address=None)
2013-01-10T18:42:51Z DEBUG dnszone_add(u'122.168.192.in-addr.arpa.', idnssoamname=u'rhel6-2.testrelm.co
m.', idnssoarname=u'hostmaster.testrelm.com.', idnssoaserial=1357843371, idnssoarefresh=3600, idnssoare
try=900, idnssoaexpire=1209600, idnssoaminimum=3600, idnsupdatepolicy=u'grant TESTRELM.COM krb5-subdoma
in 122.168.192.in-addr.arpa. PTR;', idnsallowdynupdate=True, idnsallowquery=u'any;', idnsallowtransfer=
u'none;', force=True, ip_address=None, all=False, raw=False)
2013-01-10T18:42:51Z DEBUG raw: dnsrecord_add(u'122.168.192.in-addr.arpa.', u'@', nsrecord=u'rhel6-2.te
strelm.com.', force=True)
2013-01-10T18:42:51Z DEBUG dnsrecord_add(u'122.168.192.in-addr.arpa.', u'@', a_extra_create_reverse=Fal
se, aaaa_extra_create_reverse=False, nsrecord=(u'rhel6-2.testrelm.com.',), force=True, structured=False
, all=False, raw=False)
2013-01-10T18:42:51Z DEBUG   duration: 0 seconds

And I can see it in ldap on replica:

[root@rhel6-2 shm]# ldapsearch -h $(hostname) -xLLL -D "cn=Directory Manager" -w $ADMINPW -b dc=testrelm,dc=com idnsname=122.168.192.in-addr.arpa.
dn: idnsname=122.168.192.in-addr.arpa.,cn=dns,dc=testrelm,dc=com
idnsSOAminimum: 3600
idnsSOAexpire: 1209600
idnsSOAretry: 900
idnsSOArefresh: 3600
idnsSOAserial: 1357843373
idnsZoneActive: TRUE
nSRecord: rhel6-2.testrelm.com.
objectClass: top
objectClass: idnsrecord
objectClass: idnszone
idnsAllowTransfer: none;
idnsUpdatePolicy: grant TESTRELM.COM krb5-subdomain 122.168.192.in-addr.arpa.
 PTR;
idnsAllowQuery: any;
idnsName: 122.168.192.in-addr.arpa.
idnsSOAmName: rhel6-2.testrelm.com.
idnsSOArName: hostmaster.testrelm.com.
idnsAllowDynUpdate: TRUE

but, I cannot see it in ldap on master:
[root@rhel6-1 log]# ldapsearch -h $(hostname) -xLLL -D "cn=Directory Manager" -w $ADMINPW -b dc=testrelm,dc=com idnsname=122.168.192.in-addr.arpa.
dn: idnsname=122.168.192.in-addr.arpa.,cn=dns,dc=testrelm,dc=com
idnsSOAminimum: 3600
idnsSOAexpire: 1209600
idnsSOAretry: 900
idnsSOArefresh: 3600
idnsZoneActive: TRUE
nSRecord: rhel6-2.testrelm.com.
objectClass: top
objectClass: idnsrecord
objectClass: idnszone
idnsAllowTransfer: none;
idnsUpdatePolicy: grant TESTRELM.COM krb5-subdomain 122.168.192.in-addr.arpa.
 PTR;
idnsAllowQuery: any;
idnsName: 122.168.192.in-addr.arpa.
idnsSOAmName: rhel6-2.testrelm.com.
idnsSOArName: hostmaster.testrelm.com.
idnsAllowDynUpdate: TRUE

Also, I did confirm that I could reproduce it (at least with those ipa-replica-install options.  So, I'll go ahead and open a bug now and we can work from that I think.


Expected results:

idnssoaserial set properly and synced across all servers.

Additional info:
Comment 1 Rob Crittenden 2013-01-10 20:22:52 UTC 
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3340
Comment 3 Scott Poore 2013-01-10 20:24:30 UTC 
missed listing version:

ipa-server-3.0.0-19.el6.x86_64
Comment 4 Martin Kosek 2013-01-11 08:27:32 UTC 
Hello Scott, DNS SOA serial is not synchronized on purpose due to the SOA serial autoincrement feature in bind-dyndb-ldap component. In order to avoid replication issues in SOA serial increments, the attribute is not replicated.

This, however, causes masters other than the one where a zone was created to miss the SOA serial attribute and fail in serving the zone:

# ipa dnszone-show example.com
  Zone name: example.com
  Authoritative nameserver: vm-037.idm.lab.bos.redhat.com.
  Administrator e-mail address: hostmaster.example.com.
  SOA refresh: 3600
  SOA retry: 900
  SOA expire: 1209600
  SOA minimum: 3600
  Active zone: TRUE
  Allow query: any;
  Allow transfer: none;

/var/log/messages:
...
Jan 11 03:17:22 vm-024 named[27579]: zone example.com/IN: has 0 SOA records
Jan 11 03:17:22 vm-024 named[27579]: zone example.com/IN: not loaded due to errors.
Jan 11 03:17:22 vm-024 named[27579]: update_zone (psearch) failed for 'idnsname=example.com,cn=dns,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com'. Zones can be outdated, run `rndc reload`: bad zone
Jan 11 03:17:22 vm-024 named[27579]: zone example.com/IN: has 0 SOA records
Jan 11 03:17:22 vm-024 named[27579]: zone example.com/IN: not loaded due to errors.
Jan 11 03:17:22 vm-024 named[27579]: update_zone (psearch) failed for 'idnsname=example.com,cn=dns,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com'. Zones can be outdated, run `rndc reload`: bad zone

# dig -t soa example.com

; <<>> DiG 9.9.2-P1-RedHat-9.9.2-5.P1.fc18 <<>> -t soa example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 26845
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.com.			IN	SOA

;; Query time: 3 msec
;; SERVER: 10.16.78.24#53(10.16.78.24)
;; WHEN: Fri Jan 11 03:22:09 2013
;; MSG SIZE  rcvd: 40

Petr, could this issue be fixed in bind-dyndb-ldap component which would be less strict to missing SOA serial attribute and ideally fill it with default value (current unix timestamp) when it is missing?

Other solution would be for IPA dnszone-add command to connect to each other replica and fill this attribute or configure the replication agreement to replicate this attribute just for the first time when the entry is created - and I don't think that either approach is an option.
Comment 5 Petr Spacek 2013-01-11 13:58:11 UTC 
First workaround:

On each IPA server run:
ldapmodify -Y GSSAPI << EOF
dn: idnsname=example.com.,cn=dns,dc=corp,dc=test
changetype: modify
add: idnsSOAserial
idnsSOAserial: 1
EOF

and then reload BIND:
rndc reload

Note: DN above have to be modified to match real installation.
Comment 6 Petr Spacek 2013-01-11 17:43:09 UTC 
Simpler workaround:

On each IPA server run:

ipa dnszone-mod --serial=1 example.com
rndc reload
Comment 7 Petr Spacek 2013-01-14 14:23:07 UTC 
Hotfix for bind-dyndb-ldap was ACKed upstream:
https://www.redhat.com/archives/freeipa-devel/2013-January/msg00070.html
Comment 9 Scott Poore 2013-01-18 04:11:23 UTC 
Should the fix here have completely prevented creating/syncing the reverse zone on the remote server?

I'm now seeing it come up locally on the newly created replica but, not on the remote master.

Well, at least partially, I can run dig and see it but, can't see it from ipa commands:

[root@rhel6-2 log]# dig @localhost 122.168.192.in-addr.arpa. ns

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6 <<>> @localhost 122.168.192.in-addr.arpa. ns
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31444
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;122.168.192.in-addr.arpa.	IN	NS

;; ANSWER SECTION:
122.168.192.in-addr.arpa. 86400	IN	NS	rhel6-2.testrelm.com.

;; ADDITIONAL SECTION:
rhel6-2.testrelm.com.	1200	IN	A	192.168.122.62

;; Query time: 0 msec
;; SERVER: ::1#53(::1)
;; WHEN: Thu Jan 17 22:06:26 2013
;; MSG SIZE  rcvd: 92

[root@rhel6-2 log]# ipa dnszone-show 122.168.192.in-addr.arpa.
ipa: ERROR: 122.168.192.in-addr.arpa.: DNS zone not found


Now, that being said, I think something else may have gone horribly wrong with my replica install:

in dirsrv errors I see this:

[17/Jan/2013:22:09:47 -0600] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/EXAMPLE.COM not found in Kerberos database)) errno 2 (No such file or directory)
[17/Jan/2013:22:09:47 -0600] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error)

That's preventing replication right?
Comment 10 Petr Spacek 2013-01-18 14:22:52 UTC 
Scott, do you have default IPA 3.0 installation, i.e. serial_autoincrement enabled in /etc/named.conf? Are both servers DNS enabled? (I.e. --setup-dns switch used on both servers?)

Could you post results from ldapsearch -Y GSSAPI -b 'idnsname=example.com, cn=dns, dc=testrelm, dc=com' from both servers?

In default install it should work as usual if all servers are IPA+DNS enabled.

Ldapsearch should shed some light to this problem.
Comment 11 Scott Poore 2013-01-18 14:42:18 UTC 
Pretty default I guess.  From history:

on rhel6-1:

ipa-server-install --setup-dns --forwarder=$DNSFORWARD --hostname=$hostname_s.$DOMAIN -r $RELM -n $DOMAIN -p $ADMINPW -P $ADMINPW -a $ADMINPW -U

echo "$SLAVEIP $SLAVE_S.$DOMAIN" >> /etc/hosts

ipa-replica-prepare -p $ADMINPW $SLAVE_S.$DOMAIN


first test on rhel6-2:

ipa-replica-install -U --setup-dns --no-forwarders -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$SLAVE_S.$DOMAIN.gpg

last test on rhel6-2"

ipa-replica-install -U --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$SLAVE_S.$DOMAIN.gpg

I see this in named.conf on both:

        arg "zone_refresh 0";
        arg "psearch yes";
        arg "serial_autoincrement yes";


############ on rhel6-1:

[root@rhel6-1 ~]#  ldapsearch -Y GSSAPI -b 'idnsname=testrelm.com, cn=dns,dc=testrelm,dc=com'
SASL/GSSAPI authentication started
SASL username: admin
SASL SSF: 56
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <idnsname=testrelm.com, cn=dns,dc=testrelm,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# testrelm.com, dns, testrelm.com
dn: idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
idnsSOAminimum: 3600
idnsSOAexpire: 1209600
idnsSOAretry: 900
idnsSOArefresh: 3600
idnsSOAserial: 1358481398
idnsZoneActive: TRUE
nSRecord: rhel6-1.testrelm.com.
objectClass: top
objectClass: idnsrecord
objectClass: idnszone
idnsAllowTransfer: none;
idnsUpdatePolicy: grant TESTRELM.COM krb5-self * A; grant TESTRELM.COM krb5-se
 lf * AAAA; grant TESTRELM.COM krb5-self * SSHFP;
idnsAllowQuery: any;
idnsName: testrelm.com
idnsSOAmName: rhel6-1.testrelm.com.
idnsSOArName: hostmaster.testrelm.com.
idnsAllowDynUpdate: TRUE

# rhel6-1, testrelm.com, dns, testrelm.com
dn: idnsname=rhel6-1,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
objectClass: top
objectClass: idnsrecord
aRecord: 192.168.122.61
idnsName: rhel6-1
sSHFPRecord: 2 1 34D46F39551DFFFB474E864021BF5CB8296081A3
sSHFPRecord: 1 1 2AC86DD5EFCE01B6E16785DF47A0316457F039B7
dNSTTL: 1200

# _ldap._tcp, testrelm.com, dns, testrelm.com
dn: idnsname=_ldap._tcp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
objectClass: top
objectClass: idnsrecord
sRVRecord: 0 100 389 rhel6-1
sRVRecord: 0 100 389 rhel6-2
idnsName: _ldap._tcp

# _kerberos, testrelm.com, dns, testrelm.com
dn: idnsname=_kerberos,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
objectClass: top
objectClass: idnsrecord
tXTRecord: TESTRELM.COM
idnsName: _kerberos

# _kerberos._tcp, testrelm.com, dns, testrelm.com
dn: idnsname=_kerberos._tcp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
objectClass: top
objectClass: idnsrecord
sRVRecord: 0 100 88 rhel6-1
sRVRecord: 0 100 88 rhel6-2
idnsName: _kerberos._tcp

# _kerberos._udp, testrelm.com, dns, testrelm.com
dn: idnsname=_kerberos._udp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
objectClass: top
objectClass: idnsrecord
sRVRecord: 0 100 88 rhel6-1
sRVRecord: 0 100 88 rhel6-2
idnsName: _kerberos._udp

# _kerberos-master._tcp, testrelm.com, dns, testrelm.com
dn: idnsname=_kerberos-master._tcp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc
 =com
objectClass: top
objectClass: idnsrecord
sRVRecord: 0 100 88 rhel6-1
sRVRecord: 0 100 88 rhel6-2
idnsName: _kerberos-master._tcp

# _kerberos-master._udp, testrelm.com, dns, testrelm.com
dn: idnsname=_kerberos-master._udp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc
 =com
objectClass: top
objectClass: idnsrecord
sRVRecord: 0 100 88 rhel6-1
sRVRecord: 0 100 88 rhel6-2
idnsName: _kerberos-master._udp

# _kpasswd._tcp, testrelm.com, dns, testrelm.com
dn: idnsname=_kpasswd._tcp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
objectClass: top
objectClass: idnsrecord
sRVRecord: 0 100 464 rhel6-1
sRVRecord: 0 100 464 rhel6-2
idnsName: _kpasswd._tcp

# _kpasswd._udp, testrelm.com, dns, testrelm.com
dn: idnsname=_kpasswd._udp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
objectClass: top
objectClass: idnsrecord
sRVRecord: 0 100 464 rhel6-1
sRVRecord: 0 100 464 rhel6-2
idnsName: _kpasswd._udp

# _ntp._udp, testrelm.com, dns, testrelm.com
dn: idnsname=_ntp._udp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
objectClass: top
objectClass: idnsrecord
sRVRecord: 0 100 123 rhel6-1
sRVRecord: 0 100 123 rhel6-2
idnsName: _ntp._udp

# rhel6-2, testrelm.com, dns, testrelm.com
dn: idnsname=rhel6-2,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
objectClass: top
objectClass: idnsrecord
aRecord: 192.168.122.62
idnsName: rhel6-2

# search result
search: 4
result: 0 Success

# numResponses: 13
# numEntries: 12


############ on rhel6-2:

[root@rhel6-2 ~]# ldapsearch -Y GSSAPI -b 'idnsname=testrelm.com, cn=dns,dc=testrelm,dc=com'
SASL/GSSAPI authentication started
SASL username: admin
SASL SSF: 56
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <idnsname=testrelm.com, cn=dns,dc=testrelm,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# testrelm.com, dns, testrelm.com
dn: idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
idnsSOAminimum: 3600
idnsSOAexpire: 1209600
idnsSOAretry: 900
idnsSOArefresh: 3600
idnsSOAserial: 1358481945
idnsAllowDynUpdate: TRUE
idnsSOArName: hostmaster.testrelm.com.
idnsSOAmName: rhel6-1.testrelm.com.
idnsName: testrelm.com
idnsAllowQuery: any;
idnsUpdatePolicy: grant TESTRELM.COM krb5-self * A; grant TESTRELM.COM krb5-se
 lf * AAAA; grant TESTRELM.COM krb5-self * SSHFP;
idnsAllowTransfer: none;
objectClass: top
objectClass: idnsrecord
objectClass: idnszone
nSRecord: rhel6-1.testrelm.com.
nSRecord: rhel6-2.testrelm.com.
idnsZoneActive: TRUE

# rhel6-1, testrelm.com, dns, testrelm.com
dn: idnsname=rhel6-1,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
dNSTTL: 1200
sSHFPRecord: 2 1 34D46F39551DFFFB474E864021BF5CB8296081A3
sSHFPRecord: 1 1 2AC86DD5EFCE01B6E16785DF47A0316457F039B7
idnsName: rhel6-1
aRecord: 192.168.122.61
objectClass: top
objectClass: idnsrecord

# _ldap._tcp, testrelm.com, dns, testrelm.com
dn: idnsname=_ldap._tcp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
idnsName: _ldap._tcp
sRVRecord: 0 100 389 rhel6-1
sRVRecord: 0 100 389 rhel6-2
objectClass: top
objectClass: idnsrecord

# _kerberos, testrelm.com, dns, testrelm.com
dn: idnsname=_kerberos,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
idnsName: _kerberos
tXTRecord: TESTRELM.COM
objectClass: top
objectClass: idnsrecord

# _kerberos._tcp, testrelm.com, dns, testrelm.com
dn: idnsname=_kerberos._tcp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
idnsName: _kerberos._tcp
sRVRecord: 0 100 88 rhel6-1
sRVRecord: 0 100 88 rhel6-2
objectClass: top
objectClass: idnsrecord

# _kerberos._udp, testrelm.com, dns, testrelm.com
dn: idnsname=_kerberos._udp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
idnsName: _kerberos._udp
sRVRecord: 0 100 88 rhel6-1
sRVRecord: 0 100 88 rhel6-2
objectClass: top
objectClass: idnsrecord

# _kerberos-master._tcp, testrelm.com, dns, testrelm.com
dn: idnsname=_kerberos-master._tcp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc
 =com
idnsName: _kerberos-master._tcp
sRVRecord: 0 100 88 rhel6-1
sRVRecord: 0 100 88 rhel6-2
objectClass: top
objectClass: idnsrecord

# _kerberos-master._udp, testrelm.com, dns, testrelm.com
dn: idnsname=_kerberos-master._udp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc
 =com
idnsName: _kerberos-master._udp
sRVRecord: 0 100 88 rhel6-1
sRVRecord: 0 100 88 rhel6-2
objectClass: top
objectClass: idnsrecord

# _kpasswd._tcp, testrelm.com, dns, testrelm.com
dn: idnsname=_kpasswd._tcp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
idnsName: _kpasswd._tcp
sRVRecord: 0 100 464 rhel6-1
sRVRecord: 0 100 464 rhel6-2
objectClass: top
objectClass: idnsrecord

# _kpasswd._udp, testrelm.com, dns, testrelm.com
dn: idnsname=_kpasswd._udp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
idnsName: _kpasswd._udp
sRVRecord: 0 100 464 rhel6-1
sRVRecord: 0 100 464 rhel6-2
objectClass: top
objectClass: idnsrecord

# _ntp._udp, testrelm.com, dns, testrelm.com
dn: idnsname=_ntp._udp,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
idnsName: _ntp._udp
sRVRecord: 0 100 123 rhel6-1
sRVRecord: 0 100 123 rhel6-2
objectClass: top
objectClass: idnsrecord

# rhel6-2, testrelm.com, dns, testrelm.com
dn: idnsname=rhel6-2,idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
dNSTTL: 1200
sSHFPRecord: 2 1 34D46F39551DFFFB474E864021BF5CB8296081A3
sSHFPRecord: 1 1 2AC86DD5EFCE01B6E16785DF47A0316457F039B7
idnsName: rhel6-2
aRecord: 192.168.122.62
objectClass: top
objectClass: idnsrecord

# search result
search: 4
result: 0 Success

# numResponses: 13
# numEntries: 12
[root@rhel6-2 ~]# 

############# 

Also, checking replication:

[root@rhel6-1 ~]# ipa-replica-manage list -v $(hostname)
rhel6-2.testrelm.com: replica
  last init status: 0 Total update succeeded
  last init ended: 2013-01-18 03:57:03+00:00
  last update status: 0 Replica acquired successfully: Incremental update succeeded
  last update ended: 2013-01-18 04:21:33+00:00


[root@rhel6-2 ~]# ipa-replica-manage list -v $(hostname)
rhel6-1.testrelm.com: replica
  last init status: None
  last init ended: None
  last update status: -2  - LDAP error: Local error
  last update ended: None
Comment 12 Petr Spacek 2013-01-18 15:31:37 UTC 
Ah, we will need LDIF for zone 122.168.192.in-addr.arpa. , i.e. ldapsearch -Y GSSAPI -b 'idnsname=122.168.192.in-addr.arpa., cn=dns,dc=testrelm,dc=com' should produce the right output.

Please try to cut the last dot from '122.168.192.in-addr.arpa.' if ldapsearch produces empty LDIF.

I'm sorry for misunderstanding.
Comment 13 Scott Poore 2013-01-18 15:54:06 UTC 
Petr,

We found the problem.   It's DNS resolution with my test specifically causing the issue.

I had deleted the 122.168.192.in-addr.arpa. zone to try to "simulate" the ipa-replica-install for a new network zone.  To see it create it properly.  

Well, apparently somewhere along the line, kerberos on second server was trying to lookup IP for first server for ldap (I think).  

It was getting back the rhel6-1.example.com from dnsmasq on my KVM host because I had it as the forwarder from the initial ipa-server-install.  And that was apparently causing havoc...

Ok...trying something else to verify..will post back when I have that.

Thanks for the help,
Scott
Comment 14 Scott Poore 2013-01-18 16:07:46 UTC 
added Master info to replica's /etc/hosts and reproducing that way is working again.  That's what I missed.  I thought it had worked in the past.

So, I'll re-run cleanly from scratch and then post verification for this and the other two related (bug #894143 and bug #895083).

Thanks again
Comment 15 Scott Poore 2013-01-18 17:01:34 UTC 
Verified.

Version ::

ipa-server-3.0.0-22.el6.x86_64

Manual Test Results ::


On MASTER:

[root@rhel6-1 ~]# ipa dnszone-del 122.168.192.in-addr.arpa.

[root@rhel6-1 ~]# ipa-replica-prepare -p $ADMINPW rhel6-2.testrelm.com
Preparing replica for rhel6-2.testrelm.com from rhel6-1.testrelm.com
Creating SSL certificate for the Directory Server
Creating SSL certificate for the dogtag Directory Server
Creating SSL certificate for the Web Server
Exporting RA certificate
Copying additional files
Finalizing configuration
Packaging replica information into /var/lib/ipa/replica-info-rhel6-2.testrelm.com.gpg

On REPLICA:

[root@rhel6-2 ~]# sftp rhel6-1.testrelm.com:/var/lib/ipa/replica-info-rhel6-2.testrelm.com.gpg /var/lib/ipa
Connecting to rhel6-1.testrelm.com...
The authenticity of host 'rhel6-1.testrelm.com (192.168.122.61)' can't be established.
RSA key fingerprint is 5f:a4:46:34:99:80:f7:8b:1b:76:f0:e7:d6:97:25:24.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rhel6-1.testrelm.com,192.168.122.61' (RSA) to the list of known hosts.
root.com's password: 
Fetching /var/lib/ipa/replica-info-rhel6-2.testrelm.com.gpg to /var/lib/ipa/replica-info-rhel6-2.testrelm.com.gpg
/var/lib/ipa/replica-info-rhel6-2.testrelm.com.gpg                   100%   35KB  35.3KB/s   00:00    

[root@rhel6-2 ~]# ipa-replica-install -U --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /var/lib/ipa/replica-info-rhel6-2.testrelm.com.gpg
Run connection check to master
Check connection from replica to remote master 'rhel6-1.testrelm.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

The following list of ports use UDP protocol and would need to be
checked manually:
   Kerberos KDC: UDP (88): SKIPPED
   Kerberos Kpasswd: UDP (464): SKIPPED

Connection from replica to master is OK.
Start listening on required ports for remote master check
Get credentials to log in to remote master
Execute check on remote master
Check connection from master to remote replica 'rhel6-2.testrelm.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

Connection from master to replica is OK.

Connection check OK
Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv): Estimated time 1 minute
  [1/31]: creating directory server user
  [2/31]: creating directory server instance
  [3/31]: adding default schema
  [4/31]: enabling memberof plugin
  [5/31]: enabling winsync plugin
  [6/31]: configuring replication version plugin
  [7/31]: enabling IPA enrollment plugin
  [8/31]: enabling ldapi
  [9/31]: disabling betxn plugins
  [10/31]: configuring uniqueness plugin
  [11/31]: configuring uuid plugin
  [12/31]: configuring modrdn plugin
  [13/31]: enabling entryUSN plugin
  [14/31]: configuring lockout plugin
  [15/31]: creating indices
  [16/31]: enabling referential integrity plugin
  [17/31]: configuring ssl for ds instance
  [18/31]: configuring certmap.conf
  [19/31]: configure autobind for root
  [20/31]: configure new location for managed entries
  [21/31]: restarting directory server
  [22/31]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress
Update in progress
Update in progress
Update in progress
Update in progress
Update in progress
Update in progress
Update in progress
Update succeeded
  [23/31]: adding replication acis
  [24/31]: setting Auto Member configuration
  [25/31]: enabling S4U2Proxy delegation
  [26/31]: initializing group membership
  [27/31]: adding master entry
  [28/31]: configuring Posix uid/gid generation
  [29/31]: enabling compatibility plugin
  [30/31]: tuning directory server
  [31/31]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc): Estimated time 30 seconds
  [1/9]: adding sasl mappings to the directory
  [2/9]: writing stash file from DS
  [3/9]: configuring KDC
  [4/9]: creating a keytab for the directory
  [5/9]: creating a keytab for the machine
  [6/9]: adding the password extension to the directory
  [7/9]: enable GSSAPI for replication
  [8/9]: starting the KDC
  [9/9]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring ipa_memcached
  [1/2]: starting ipa_memcached 
  [2/2]: configuring ipa_memcached to start on boot
Done configuring ipa_memcached.
Configuring the web interface (httpd): Estimated time 1 minute
  [1/12]: setting mod_nss port to 443
  [2/12]: setting mod_nss password file
  [3/12]: enabling mod_nss renegotiate
  [4/12]: adding URL rewriting rules
  [5/12]: configuring httpd
  [6/12]: setting up ssl
  [7/12]: publish CA cert
  [8/12]: creating a keytab for httpd
  [9/12]: clean up any existing httpd ccache
  [10/12]: configuring SELinux for httpd
  [11/12]: restarting httpd
  [12/12]: configuring httpd to start on boot
Done configuring the web interface (httpd).
Applying LDAP updates
Restarting the directory server
Restarting the KDC
Using reverse zone 122.168.192.in-addr.arpa.
Configuring DNS (named)
  [1/8]: adding NS record to the zone
  [2/8]: setting up reverse zone
  [3/8]: setting up our own record
  [4/8]: setting up kerberos principal
  [5/8]: setting up named.conf
  [6/8]: restarting named
  [7/8]: configuring named to start on boot
  [8/8]: changing resolv.conf to point to ourselves
Done configuring DNS (named).

Global DNS configuration in LDAP server is empty
You can use 'dnsconfig-mod' command to set global DNS options that
would override settings in local named.conf files

Restarting the web server

[root@rhel6-2 ~]# kinit admin
Password for admin: 

[root@rhel6-2 ~]# ipa dnszone-show 122.168.192.in-addr.arpa.
  Zone name: 122.168.192.in-addr.arpa.
  Authoritative nameserver: rhel6-2.testrelm.com.
  Administrator e-mail address: hostmaster.testrelm.com.
  SOA serial: 1358527393
  SOA refresh: 3600
  SOA retry: 900
  SOA expire: 1209600
  SOA minimum: 3600
  Active zone: TRUE
  Allow query: any;
  Allow transfer: none;

On MASTER:

[root@rhel6-1 ~]# ipa dnszone-show 122.168.192.in-addr.arpa.
  Zone name: 122.168.192.in-addr.arpa.
  Authoritative nameserver: rhel6-2.testrelm.com.
  Administrator e-mail address: hostmaster.testrelm.com.
  SOA serial: 1358527393
  SOA refresh: 3600
  SOA retry: 900
  SOA expire: 1209600
  SOA minimum: 3600
  Active zone: TRUE
  Allow query: any;
  Allow transfer: none;

I see the SOA serial there on MASTER so looks like this is good.
Comment 17 errata-xmlrpc 2013-02-21 09:32:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0528.html