Created attachment 516959 [details]
dns config tgz file

Description of problem:
========================
With a default matahari configuration and a resolvable _matahari DNS SRV record, the matahari agents fail to find the broker specified through DNS.  It was only when I specified MATAHARI_BROKER=domain_name (in this case, test.com) would the agent connect to the broker.  It seems like the failover to query for a DNS SRV fails to determine the domain name and ends up querying for _matahari._tcp.127.0.0.1 instead of _matahari._tcp.test.com.  This is seen in the DNS server's /var/log/messages: 

named[13961]: client 10.16.44.45#57274: query (cache) '_matahari._tcp.127.0.0.1/SRV/IN' denied

I setup three boxes, one for the dns server, one for the broker, and one for the agent.  DNS configuration is attached but it was a simple configuration which had both forward & reverse lookups for the broker and agent hosts as well as a DNS SRV record:

    _matahari._tcp			86400 IN SRV 1 1 49000 broker.test.com.

I updated the broker and agent DNS config and made sure `hostname -f` returned correctly (like broker.test.com) and `domainname` returned test.com.  Both the broker and the agent could resolve test.com: 

    [root@ibm-hs21-03 init.d]# nslookup test.com
    Server:		10.16.65.39
    Address:	10.16.65.39#53

    Name:	test.com
    Address: 10.16.65.39
    
    [root@ibm-hs21-03 init.d]# hostname
    agent
    [root@ibm-hs21-03 init.d]# domainname
    test.com
    [root@ibm-hs21-03 init.d]# hostname -f
    agent.test.com

Both could also query DNS for the matahari service:

    [root@ibm-hs21-03 init.d]# nslookup
    > set type=SRV
    > _matahari._tcp
    Server:		10.16.65.39
    Address:	10.16.65.39#53

    _matahari._tcp.test.com	service = 1 1 49000 broker.test.com.


I spoke to Adam who discovered that when starting the agent without a broker, the same thing seemed to happen.  Fairly sure this is related if not the same issue.

[root@ibm-hs21-03 init.d]# QPID_LOG_ENABLE=debug+ matahari-qmf-networkd
2011-08-05 16:07:46 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2011-08-05 16:07:46 debug Created connection amqp:tcp:127.0.0.1:5672 with {}
2011-08-05 16:07:46 debug Created connection amqp:tcp:localhost:49000 with {reconnect:False}
2011-08-05 16:07:46 info Trying to connect to amqp:tcp:localhost:49000...
2011-08-05 16:07:46 debug Created IO thread: 0
2011-08-05 16:07:46 debug TCPConnector created for 0-10
2011-08-05 16:07:46 info Connection  connected to tcp:localhost:49000
2011-08-05 16:07:46 warning Connect failed: Connection refused
2011-08-05 16:07:46 warning Connection  closed
2011-08-05 16:07:46 debug Exception constructed: Connection  closed
2011-08-05 16:07:46 info Failed to connect to amqp:tcp:localhost:49000: Connection  closed
2011-08-05 16:07:46 debug Created connection amqp:tcp::49000 with {reconnect:False}
2011-08-05 16:07:46 info Trying to connect to amqp:tcp::49000...
2011-08-05 16:07:46 debug Exception constructed: Invalid URL: amqp:tcp::49000 (qpid/Url.cpp:237)
2011-08-05 16:07:46 debug Created connection amqp:tcp::49000amqp:tcp:localhost:49000 with {reconnect:False}
2011-08-05 16:07:46 info Trying to connect to amqp:tcp::49000amqp:tcp:localhost:49000...
2011-08-05 16:07:46 debug Exception constructed: Invalid URL: amqp:tcp::49000amqp:tcp:localhost:49000 (qpid/Url.cpp:237)


Version-Release number of selected component (if applicable):
matahari-0.4.2-2.el6.x86_64
matahari-agent-lib-0.4.2-2.el6.x86_64
matahari-broker-0.4.2-2.el6.x86_64
matahari-host-0.4.2-2.el6.x86_64
matahari-lib-0.4.2-2.el6.x86_64
matahari-network-0.4.2-2.el6.x86_64
matahari-service-0.4.2-2.el6.x86_64
matahari-sysconfig-0.4.2-2.el6.x86_64
qpid-cpp-client-0.10-6.el6.x86_64
qpid-cpp-client-devel-0.10-6.el6.x86_64
qpid-cpp-client-ssl-0.10-6.el6.x86_64
qpid-cpp-server-0.10-6.el6.x86_64
qpid-cpp-server-ssl-0.10-6.el6.x86_64
qpid-qmf-0.10-6.el6.x86_64
sigar-1.6.5-0.1.git833ca18.el6.x86_64


How reproducible:
100%

Steps to Reproduce:
1.  setup attached DNS config (obviously you will need to update appropriate IP addresses and associated subnet zones depending on hosts used)
2.  configure broker to use DNS
3.  configure agent to use DNS
4.  verify hostname, domainname, hostname -f returns correctly
5.  verify nslookup test.com resolves
6.  verify the matahari service record returns correctly (example above)
7.  verify default agent config /etc/sysconfig/matahari, MATAHARI_BROKER=127.0.0.1
8.  start matahari-broker on broker server
9.  start matahari-agent on agent system
10.  see agent's /var/log/messages not finding a broker, keeps retrying
11. see dns's /var/log/mesages error _matahari._tcp.127.0.0.1/SRV/IN' denied
12. stop matahari agent
13. set /etc/sysconfig/matahari, MATAHARI_BROKER=test.com
14. start matahari-agent on agent system
15. see agent's /var/log/messages, agent connects to broker