Bug 921057 - The agent is not able to poll the server on IPv6
Summary: The agent is not able to poll the server on IPv6
Keywords:
Status: NEW
Alias: None
Product: RHQ Project
Classification: Other
Component: Agent, Communications Subsystem
Version: 4.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Nobody
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 919199
TreeView+ depends on / blocked
 
Reported: 2013-03-13 11:53 UTC by Filip Brychta
Modified: 2022-04-23 08:28 UTC (History)
4 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed:
Embargoed:


Attachments (Terms of Use)
rhq-server.properties (11.81 KB, application/octet-stream)
2013-03-13 11:53 UTC, Filip Brychta
no flags Details
agent log (46.54 KB, text/x-log)
2013-03-13 11:53 UTC, Filip Brychta
no flags Details

Description Filip Brychta 2013-03-13 11:53:26 UTC
Created attachment 709544 [details]
rhq-server.properties

Description of problem:
I have successfully configured the RHQ server on IPv6 but the agent is not able to poll the server so the registration process fails and the agent gets stuck on "The agent will now wait until it has registered with the server..."

Version-Release number of selected component (if applicable):
Version: 4.7.0-SNAPSHOT
Build Number: 2174d4c

How reproducible:
Always

Steps to Reproduce:
1. configure the server to use IPv6:
  a) change the rhq-server.sh to use following java opts -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true for starting the server
  b) use previous java opts for the rhq-installer.sh as well
  c) prepare the rhq-server.properties (look at attached example)
  d) change the rhq-server.sh - line 471 should look like this:
export RHQ_SERVER_HOME; ${RHQ_SERVER_HOME}/bin/rhq-installer.sh --reconfig --host=::1 &
(i'm not sure if this is necessary)
  e) run the server - ./rhq-server.sh start
  f) run the installer - ./rhq-installer.sh --host=::1       

2. configure the agent to use IPv6:
  a)change the rhq-agent.sh to use following java opts -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true
  b) enable debug mode
  c) run the agent with a clean config - ./rhq-agent.sh -L
  d) use IPv6  

  
Actual results:
the agent gets stuck on "The agent will now wait until it has registered with the server..."
And the agent log ends with:
2013-03-13 06:21:53,358 DEBUG [RHQ Server Polling Thread] (jboss.remoting.transport.http.HTTPClientInvoker)- Unable to retrieve response message
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
        at java.net.Socket.connect(Socket.java:546)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:173)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:409)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:240)
        at sun.net.www.http.HttpClient.New(HttpClient.java:321)
        at sun.net.www.http.HttpClient.New(HttpClient.java:338)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:935)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:876)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:801)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1139)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
        at java.net.HttpURLConnection.getResponseMessage(HttpURLConnection.java:463)
        at org.jboss.remoting.transport.http.HTTPClientInvoker.useHttpURLConnection(HTTPClientInvoker.java:340)
        at org.jboss.remoting.transport.http.HTTPClientInvoker.transport(HTTPClientInvoker.java:137)
        at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
        at org.jboss.remoting.Client.invoke(Client.java:1634)
        at org.jboss.remoting.Client.invoke(Client.java:548)
        at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.rawSend(JBossRemotingRemoteCommunicator.java:514)
        at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.sendWithoutCallbacks(JBossRemotingRemoteCommunicator.java:456)
        at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.sendWithoutInitializeCallback(JBossRemotingRemoteCommunicator.java:475)
        at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.send(JBossRemotingRemoteCommunicator.java:496)
        at org.rhq.enterprise.communications.command.client.AbstractCommandClient.invoke(AbstractCommandClient.java:143)
        at org.rhq.enterprise.communications.command.client.ClientCommandSender.send(ClientCommandSender.java:1091)
        at org.rhq.enterprise.communications.command.client.ServerPollingThread.run(ServerPollingThread.java:115)
2013-03-13 06:21:53,363 DEBUG [RHQ Server Polling Thread] (enterprise.communications.command.client.ServerPollingThread)- {ServerPollingThread.server-poll-failure}Failed to successfully poll the server. This is normally due to the server not being up yet. You can usually ignore this message since it will be tried again later, however, you should ensure this failure was not really caused by a misconfiguration. Cause: org.jboss.remoting.CannotConnectException:Can not connect http client invoker. Connection refused. -> java.net.ConnectException:Connection refused



Expected results:
Agent is registered with the server.




Additional info:
The cause could be that rhq-agent/lib/jboss-remoting-2.2.2.SP8.jar doesn't support the IPv6 (this should be confirmed by someone)

I noticed in the agent's log following:
2013-03-13 06:21:53,229 DEBUG [main] (org.rhq.enterprise.communications.ServiceContainer)- {ServiceContainer.connector-uri}Connector is using locator URI of [InvokerLocator [socket://10.16.23.190/?backlog=200&clientMaxPoolSize=304&enableTcpNoDelay=true&maxPoolSize=303&numAcceptThreads=1&rhq.communications.connector.rhqtype=agent&socketTimeout=60000]]

there is a IPv4 (10.16.23.190) but ~/.java/.userPrefs/rhq-agent/default/prefs.xml contains only IPv6. We suspect that jboss-remoting library changed IPv6 to IPv4

Comment 1 Filip Brychta 2013-03-13 11:53:50 UTC
Created attachment 709545 [details]
agent log

Comment 2 John Mazzitelli 2013-03-13 12:28:21 UTC
That was the first thing I thought of - does JBoss/Remoting 2 support IPv6? I don't know.

The thing that I dn't get is that log message with the IPv4 address. Somehow, someway, the agent's address is getting reset to that.

Can you attach the FULL agent log (with debug enabled) from start to finish?

In otherwords, start with a clean agent, have debug enabled (RHQ_AGENT_DEBUG=true), and do that again, then attach agent.log.

Also, attach the output of "getconfig" to that BZ as well

Comment 3 John Mazzitelli 2013-03-13 12:29:50 UTC
opps... duh... didn't notice you already attached agent.log :)

can you post the "getconfig" output, too?

Comment 4 Filip Brychta 2013-03-13 12:51:15 UTC
attaching the agent's config:

shutdown> getconfig 
rhq.agent.agent-update.enabled=true
rhq.agent.client.command-preprocessors=org.rhq.enterprise.agent.SecurityTokenCommandPreprocessor:org.rhq.enterprise.agent.ExternalizableStrategyCommandPreprocessor
rhq.agent.client.command-spool-file.compressed=true
rhq.agent.client.command-spool-file.name=command-spool.dat
rhq.agent.client.command-spool-file.params=10000000:75
rhq.agent.client.command-timeout-msecs=600000
rhq.agent.client.max-concurrent=5
rhq.agent.client.max-retries=10
rhq.agent.client.queue-size=50000
rhq.agent.client.queue-throttling=200:2000
rhq.agent.client.retry-interval-msecs=15000
rhq.agent.client.send-throttling=100:1000
rhq.agent.client.server-polling-interval-msecs=60000
rhq.agent.configuration-schema-version=6
rhq.agent.configuration-setup-flag=true
rhq.agent.data-directory=data
rhq.agent.disable-native-system=false
rhq.agent.name=last-rhq-fbrychta-5m.bc.jonqe.lab.eng.bos.redhat.com
rhq.agent.plugins.directory=plugins
rhq.agent.primary-server-switchover-check-interval-msecs=3600000
rhq.agent.register-with-server-at-startup=true
rhq.agent.server-auto-detection=false
rhq.agent.server.alias=rhqserver
rhq.agent.server.bind-address=2620:52:0:1017:21a:4aff:fe10:17d9
rhq.agent.server.bind-port=7080
rhq.agent.server.transport=servlet
rhq.agent.server.transport-params=/jboss-remoting-servlet-invoker/ServerInvokerServlet
rhq.agent.test-failover-list-at-startup=true
rhq.agent.update-plugins-at-startup=true
rhq.agent.vm-health-check.interval-msecs=5000
rhq.agent.vm-health-check.low-heap-mem-threshold=0.90
rhq.agent.vm-health-check.low-nonheap-mem-threshold=0.90
rhq.agent.wait-for-server-at-startup-msecs=60000
rhq.communications.command-service-directory.allow-dynamic-discovery=true
rhq.communications.command-services=org.rhq.enterprise.communications.command.impl.echo.server.EchoCommandService, org.rhq.enterprise.communications.command.impl.identify.server.IdentifyCommandService
rhq.communications.configuration-schema-version=1
rhq.communications.connector.bind-address=2620:52:0:1017:21a:4aff:fe10:17d9
rhq.communications.connector.bind-port=16163
rhq.communications.connector.rhqtype=agent
rhq.communications.connector.transport=socket
rhq.communications.data-directory=data
rhq.communications.global-concurrency-limit=-1
rhq.communications.multicast-detector.bind-address=0.0.0.0
rhq.communications.multicast-detector.default-time-delay=5000
rhq.communications.multicast-detector.enabled=false
rhq.communications.multicast-detector.heartbeat-time-delay=1000
rhq.communications.multicast-detector.multicast-address=224.16.16.16
rhq.communications.multicast-detector.port=16162
rhq.communications.remote-pojos=org.rhq.enterprise.communications.PingImpl:org.rhq.enterprise.communications.Ping
rhq.communications.remote-stream-max-idle-time-msecs=300000

Comment 5 Filip Brychta 2013-03-13 14:06:37 UTC
I tried it again with all IPs surrounded by brackets (in the agent and server config) and it works. The problem with brackets is probably related to https://issues.jboss.org/browse/JBREM-1164 and brackets in configuration should not be required as soon as the jboss-remoting will be upgraded. (thanks mazz)

Comment 6 Mike Foley 2013-03-13 14:13:05 UTC
this might make a good documentation issue.  adding sunny to notify list


Note You need to log in before you can comment on or make changes to this bug.