Description of problem:
Running "puppet facts" from the cli on an overcloud node takes around 80 seconds to complete and "normal" would be taking 2-3 seconds.
call is taking a really long time due to gathering ipa facts via /usr/share/openstack-puppet/modules/ipaclient/lib/facter/ipa_facts.rb caused by a DNS TXT lookup for "_kerberos" (which doesn't exist in DNS)
Looking in /usr/share/ruby/resolv.rb, the default timeout seems to be:
# [ 5, second = 5 * 2 / nameserver_count, 2 * second, 4 * second ]
You can see the difference in timing when the ipa facts are pre-populated (the DNS lookup is skipped) or not:
[root@]# time FACTER_ipa_domain='' FACTER_ipa_server='' puppet facts >/dev/null
real 0m2.367s
user 0m1.746s
sys 0m0.392s
[root@]# time puppet facts >/dev/null
real 1m22.435s
user 0m1.735s
sys 0m0.428s
In the recent versions of OSP, we're running the fact execution upfront and actually caching them. In general they aren't run as much as they used to be (it used to be a lot). Since the issue seems to be in core ruby, it's likely we should kick this over to the ruby maintainers for RHEL.