Bug 921057

Summary: The agent is not able to poll the server on IPv6
Product: [Other] RHQ Project Reporter: Filip Brychta <fbrychta>
Component: Agent, Communications SubsystemAssignee: Nobody <nobody>
Status: NEW --- QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.6CC: hrupp, kejohnso, mazz, theute
Target Milestone: ---Keywords: Documentation
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 919199    
Attachments:
Description Flags
rhq-server.properties
none
agent log none

Description Filip Brychta 2013-03-13 11:53:26 UTC
Created attachment 709544 [details]
rhq-server.properties

Description of problem:
I have successfully configured the RHQ server on IPv6 but the agent is not able to poll the server so the registration process fails and the agent gets stuck on "The agent will now wait until it has registered with the server..."

Version-Release number of selected component (if applicable):
Version: 4.7.0-SNAPSHOT
Build Number: 2174d4c

How reproducible:
Always

Steps to Reproduce:
1. configure the server to use IPv6:
  a) change the rhq-server.sh to use following java opts -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true for starting the server
  b) use previous java opts for the rhq-installer.sh as well
  c) prepare the rhq-server.properties (look at attached example)
  d) change the rhq-server.sh - line 471 should look like this:
export RHQ_SERVER_HOME; ${RHQ_SERVER_HOME}/bin/rhq-installer.sh --reconfig --host=::1 &
(i'm not sure if this is necessary)
  e) run the server - ./rhq-server.sh start
  f) run the installer - ./rhq-installer.sh --host=::1       

2. configure the agent to use IPv6:
  a)change the rhq-agent.sh to use following java opts -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true
  b) enable debug mode
  c) run the agent with a clean config - ./rhq-agent.sh -L
  d) use IPv6  

  
Actual results:
the agent gets stuck on "The agent will now wait until it has registered with the server..."
And the agent log ends with:
2013-03-13 06:21:53,358 DEBUG [RHQ Server Polling Thread] (jboss.remoting.transport.http.HTTPClientInvoker)- Unable to retrieve response message
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
        at java.net.Socket.connect(Socket.java:546)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:173)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:409)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:240)
        at sun.net.www.http.HttpClient.New(HttpClient.java:321)
        at sun.net.www.http.HttpClient.New(HttpClient.java:338)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:935)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:876)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:801)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1139)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
        at java.net.HttpURLConnection.getResponseMessage(HttpURLConnection.java:463)
        at org.jboss.remoting.transport.http.HTTPClientInvoker.useHttpURLConnection(HTTPClientInvoker.java:340)
        at org.jboss.remoting.transport.http.HTTPClientInvoker.transport(HTTPClientInvoker.java:137)
        at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
        at org.jboss.remoting.Client.invoke(Client.java:1634)
        at org.jboss.remoting.Client.invoke(Client.java:548)
        at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.rawSend(JBossRemotingRemoteCommunicator.java:514)
        at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.sendWithoutCallbacks(JBossRemotingRemoteCommunicator.java:456)
        at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.sendWithoutInitializeCallback(JBossRemotingRemoteCommunicator.java:475)
        at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.send(JBossRemotingRemoteCommunicator.java:496)
        at org.rhq.enterprise.communications.command.client.AbstractCommandClient.invoke(AbstractCommandClient.java:143)
        at org.rhq.enterprise.communications.command.client.ClientCommandSender.send(ClientCommandSender.java:1091)
        at org.rhq.enterprise.communications.command.client.ServerPollingThread.run(ServerPollingThread.java:115)
2013-03-13 06:21:53,363 DEBUG [RHQ Server Polling Thread] (enterprise.communications.command.client.ServerPollingThread)- {ServerPollingThread.server-poll-failure}Failed to successfully poll the server. This is normally due to the server not being up yet. You can usually ignore this message since it will be tried again later, however, you should ensure this failure was not really caused by a misconfiguration. Cause: org.jboss.remoting.CannotConnectException:Can not connect http client invoker. Connection refused. -> java.net.ConnectException:Connection refused



Expected results:
Agent is registered with the server.




Additional info:
The cause could be that rhq-agent/lib/jboss-remoting-2.2.2.SP8.jar doesn't support the IPv6 (this should be confirmed by someone)

I noticed in the agent's log following:
2013-03-13 06:21:53,229 DEBUG [main] (org.rhq.enterprise.communications.ServiceContainer)- {ServiceContainer.connector-uri}Connector is using locator URI of [InvokerLocator [socket://10.16.23.190/?backlog=200&clientMaxPoolSize=304&enableTcpNoDelay=true&maxPoolSize=303&numAcceptThreads=1&rhq.communications.connector.rhqtype=agent&socketTimeout=60000]]

there is a IPv4 (10.16.23.190) but ~/.java/.userPrefs/rhq-agent/default/prefs.xml contains only IPv6. We suspect that jboss-remoting library changed IPv6 to IPv4

Comment 1 Filip Brychta 2013-03-13 11:53:50 UTC
Created attachment 709545 [details]
agent log

Comment 2 John Mazzitelli 2013-03-13 12:28:21 UTC
That was the first thing I thought of - does JBoss/Remoting 2 support IPv6? I don't know.

The thing that I dn't get is that log message with the IPv4 address. Somehow, someway, the agent's address is getting reset to that.

Can you attach the FULL agent log (with debug enabled) from start to finish?

In otherwords, start with a clean agent, have debug enabled (RHQ_AGENT_DEBUG=true), and do that again, then attach agent.log.

Also, attach the output of "getconfig" to that BZ as well

Comment 3 John Mazzitelli 2013-03-13 12:29:50 UTC
opps... duh... didn't notice you already attached agent.log :)

can you post the "getconfig" output, too?

Comment 4 Filip Brychta 2013-03-13 12:51:15 UTC
attaching the agent's config:

shutdown> getconfig 
rhq.agent.agent-update.enabled=true
rhq.agent.client.command-preprocessors=org.rhq.enterprise.agent.SecurityTokenCommandPreprocessor:org.rhq.enterprise.agent.ExternalizableStrategyCommandPreprocessor
rhq.agent.client.command-spool-file.compressed=true
rhq.agent.client.command-spool-file.name=command-spool.dat
rhq.agent.client.command-spool-file.params=10000000:75
rhq.agent.client.command-timeout-msecs=600000
rhq.agent.client.max-concurrent=5
rhq.agent.client.max-retries=10
rhq.agent.client.queue-size=50000
rhq.agent.client.queue-throttling=200:2000
rhq.agent.client.retry-interval-msecs=15000
rhq.agent.client.send-throttling=100:1000
rhq.agent.client.server-polling-interval-msecs=60000
rhq.agent.configuration-schema-version=6
rhq.agent.configuration-setup-flag=true
rhq.agent.data-directory=data
rhq.agent.disable-native-system=false
rhq.agent.name=last-rhq-fbrychta-5m.bc.jonqe.lab.eng.bos.redhat.com
rhq.agent.plugins.directory=plugins
rhq.agent.primary-server-switchover-check-interval-msecs=3600000
rhq.agent.register-with-server-at-startup=true
rhq.agent.server-auto-detection=false
rhq.agent.server.alias=rhqserver
rhq.agent.server.bind-address=2620:52:0:1017:21a:4aff:fe10:17d9
rhq.agent.server.bind-port=7080
rhq.agent.server.transport=servlet
rhq.agent.server.transport-params=/jboss-remoting-servlet-invoker/ServerInvokerServlet
rhq.agent.test-failover-list-at-startup=true
rhq.agent.update-plugins-at-startup=true
rhq.agent.vm-health-check.interval-msecs=5000
rhq.agent.vm-health-check.low-heap-mem-threshold=0.90
rhq.agent.vm-health-check.low-nonheap-mem-threshold=0.90
rhq.agent.wait-for-server-at-startup-msecs=60000
rhq.communications.command-service-directory.allow-dynamic-discovery=true
rhq.communications.command-services=org.rhq.enterprise.communications.command.impl.echo.server.EchoCommandService, org.rhq.enterprise.communications.command.impl.identify.server.IdentifyCommandService
rhq.communications.configuration-schema-version=1
rhq.communications.connector.bind-address=2620:52:0:1017:21a:4aff:fe10:17d9
rhq.communications.connector.bind-port=16163
rhq.communications.connector.rhqtype=agent
rhq.communications.connector.transport=socket
rhq.communications.data-directory=data
rhq.communications.global-concurrency-limit=-1
rhq.communications.multicast-detector.bind-address=0.0.0.0
rhq.communications.multicast-detector.default-time-delay=5000
rhq.communications.multicast-detector.enabled=false
rhq.communications.multicast-detector.heartbeat-time-delay=1000
rhq.communications.multicast-detector.multicast-address=224.16.16.16
rhq.communications.multicast-detector.port=16162
rhq.communications.remote-pojos=org.rhq.enterprise.communications.PingImpl:org.rhq.enterprise.communications.Ping
rhq.communications.remote-stream-max-idle-time-msecs=300000

Comment 5 Filip Brychta 2013-03-13 14:06:37 UTC
I tried it again with all IPs surrounded by brackets (in the agent and server config) and it works. The problem with brackets is probably related to https://issues.jboss.org/browse/JBREM-1164 and brackets in configuration should not be required as soon as the jboss-remoting will be upgraded. (thanks mazz)

Comment 6 Mike Foley 2013-03-13 14:13:05 UTC
this might make a good documentation issue.  adding sunny to notify list