Bug 1011045

Summary: RHEL6.3 ipa-client-install Joining realm failed
Product: Red Hat Enterprise Linux 6 Reporter: Scott Poore <spoore>
Component: ipaAssignee: Martin Kosek <mkosek>
Status: CLOSED WORKSFORME QA Contact: Namita Soman <nsoman>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.3CC: atolani, lkrispen, rcritten, spoore
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-07 15:15:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Scott Poore 2013-09-23 14:20:41 UTC
Description of problem:

I'm seeing consistent ipa-client-install failures like this:

Joining realm failed: RPC failed at server.  limits exceeded for this query
Discovery was successful!
Hostname: hp-bl460cgen8-01.testrelm.com
Realm: TESTRELM.COM
DNS Domain: testrelm.com
IPA Server: hp-dl380pgen8-02-vm-3.testrelm.com
BaseDN: dc=testrelm,dc=com


Synchronizing time with KDC...

Installation failed. Rolling back changes.
IPA client is not configured on this system.
:: [   FAIL   ] :: Running ' /usr/sbin/ipa-client-install -U --domain=$DOMAIN --realm=$RELM -p $ADMINID -w $ADMINPW --server=$MASTER' (Expected 0, got 1)
Version-Release number of selected component (if applicable):

How reproducible:
always in some test automation.

Steps to Reproduce:
1.  Install IPA Master
2.  Install IPA Replica
3.  ipa-client-install -U --domain=$DOMAIN --realm=$RELM -p $ADMINID -w $ADMINPW --server=$MASTER

Actual results:
Fails with message::

Joining realm failed: RPC failed at server.  limits exceeded for this query

Expected results:

No failure and properly joins realm.

Additional info:

Comment 1 Scott Poore 2013-09-23 14:22:06 UTC
End of ipaclient-install.log:

2013-09-23T01:52:03Z DEBUG Writing Kerberos configuration to /tmp/tmp7CKmt_:
#File modified by ipa-client-install

[libdefaults]
  default_realm = TESTRELM.COM
  dns_lookup_realm = true
  dns_lookup_kdc = true
  rdns = false
  ticket_lifetime = 24h
  forwardable = yes

[realms]
  TESTRELM.COM = {
    pkinit_anchors = FILE:/etc/ipa/ca.crt
  }

[domain_realm]
  .testrelm.com = TESTRELM.COM
  testrelm.com = TESTRELM.COM


2013-09-23T01:52:03Z DEBUG args=kinit admin
2013-09-23T01:52:03Z DEBUG stdout=Password for admin: 

2013-09-23T01:52:03Z DEBUG stderr=
2013-09-23T01:52:13Z DEBUG args=/usr/sbin/ipa-join -s hp-dl380pgen8-02-vm-3.testrelm.com -b dc=testrelm,dc=com
2013-09-23T01:52:13Z DEBUG stdout=
2013-09-23T01:52:13Z DEBUG stderr=RPC failed at server.  limits exceeded for this query

2013-09-23T01:52:13Z DEBUG args=kdestroy
2013-09-23T01:52:13Z DEBUG stdout=
2013-09-23T01:52:13Z DEBUG stderr=

Comment 3 Martin Kosek 2013-09-24 09:07:00 UTC
This rather seems as a problem with the server. Do standard ipa commands like "ipa host-find" work on the server? (i.e. on hp-dl380pgen8-02-vm-3.testrelm.com)

It might also be interesting to see 389-ds-base's access log file to see the specific searches that ended up with "limits exceeded for this query" error.

Comment 4 Scott Poore 2013-09-24 15:18:19 UTC
Yes, ipa commands ran on the Master (hp-dl380pgen8-02-vm-3.testrelm.com) right after the ipa-client-install failed.  Unfortunately, I don't have the access logs and the servers have been returned.

Also, I misread some of the results.  I must have looked at the same job twice.  I can't find where this occurred elsewhere so it's not always reproducible as I stated in the description.

Is it possible it was a network hiccup and env/server related?

Comment 5 Martin Kosek 2013-09-24 16:20:22 UTC
I think it is possible. Server may have been in a unknown state (maybe restarting? where it did not respond to LDAP requests.

I need more data to be able to track this down and fix. Especially:
* Knowing a state the server is in, trying if ipa commands work fine (we may also try cert-show command to check this is not related to PKI-CA having too long response times)
* The logs mentioned in Comment 3
* It may be also useful to try to wait e.g. 30 seconds on the client and trying to enroll again if it makes any difference.

Comment 7 Martin Kosek 2013-10-07 10:13:45 UTC
Scott, did you hit this issue again? We would need to do some investigation on the server side to be able to uncover the reason for the timeouts.

If not, I would propose to close this bug until we have more data.

Comment 8 Scott Poore 2013-10-07 15:15:25 UTC
Hey Martin,

Thanks for following up on this.

I have not seen this again and I have run a few RHEL6.3 upgrades in the last few days.  I'll go ahead and close as WORKSFORME for now.  If I am able to reproduce it in the future, I'll re-open.  I suspect, however, that this was a fluke/hiccup type issue.

Thanks again,
Scott

Comment 9 Ludwig 2014-10-16 11:47:51 UTC
I encountered this error today, when looking into DS logs I noticed that I still had ACL logging enabled and operations could take too long, especially this one:

[16/Oct/2014:13:27:16 +0200] conn=116 op=3 SRCH base="cn=ipaconfig,cn=etc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com" scope=0 filter="(objectClass=*)" attrs=ALL
[16/Oct/2014:13:27:29 +0200] conn=116 op=3 RESULT err=3 tag=101 nentries=1 etime=13

so it returned tim limit exceeded, which would match the logged error on the client side.

After setting error log level back to normal ipa-client-install succeeded.