The agent setup commands are done in the wrong order. It should ask for the server address and port before asking for the agent address and port. After taking the server adress + port it should make an unauthenticated ping command to the server, the server should return the calling address and that should be used as the default agent ip in the agent address prompt. (It would also obviously give you an immediate error if it couldn't reach the server)
see forum thread: http://forums.rhq-project.org/viewtopic.php?f=5&t=23
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-40
fixing this will go a long way towards the intuitiveness of the agent-server comm setup. over the years, since HA has been released, we've seen several customers as well as community members stumble over this. the root of the problem is caused by the fact that the server has two bind IP addresses - one which the server itself runs on, and another used for agent communications. if the agent pings the server and gets a response, the IP address the response came on should be the one used for remote communications, and the interactive setup prompt should use that value by default. this will not only save users time in the future from having to enter the IP address manually, but should cut down on the number of cases when users misconfigure this due to the dual-bind nature of the server.