Bug 1019374

Summary: server installation fails - unable to connect to 127.0.0.1:9999
Product: [JBoss] JBoss Operations Network Reporter: Armine Hovsepyan <ahovsepy>
Component: InstallerAssignee: John Mazzitelli <mazz>
Status: CLOSED NOTABUG QA Contact: Mike Foley <mfoley>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: JON 3.2CC: ahovsepy, mazz, mfoley
Target Milestone: ---   
Target Release: JON 3.2.0   
Hardware: x86_64   
OS: Solaris   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-18 12:56:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1012435    

Description Armine Hovsepyan 2013-10-15 15:07:48 UTC
Description of problem:
server installation fails - unable to connect to 127.0.0.1:9999 

Version-Release number of selected component (if applicable):
jon 3.2 er3

How reproducible:
always

Steps to Reproduce:
1. run rhqctl install --storage --start
2. run rhqctl install --server
3.

Actual results:
JON Server service is not installed. The following exeption shown:
java.net.ConnectException: JBAS012144: Could not connect to remote://127.0.0.1:9999. The connection timed out


Expected results:
jon server service installed

Additional info:
rhq-installer.log (http://d.pr/f/AGZc), rhqctl.log (http://d.pr/f/Lo26), server.log(http://d.pr/f/dKEm) and rhq-server.properties (http://d.pr/f/GHqv)

Environment: Solaris 11 x64 Sparc (dev32-01.mw.lab.eng.bos.redhat.com)

Comment 1 John Mazzitelli 2013-10-15 20:13:39 UTC
i tried this locally on my own box, and things work. I suspect this might be environmental - perhaps some odd firewall rules or something. I'll look at the logs that are attached.

Comment 2 John Mazzitelli 2013-10-15 20:30:41 UTC
this is in server.log:

09:32:39,754 INFO  [org.hornetq.jms.server] (ServerService Thread Pool -- 58) HQ121005: Invalid "host" value "0.0.0.0" detected for "netty" connector. Switching to "dev32-01". If this new address is incorrect please manually configure the connector to use the proper one.

I see your jboss.bind.address in rhq.server.proeprties is 0.0.0.0.

Can you try to use some other IP address and see what happens?

Comment 3 John Mazzitelli 2013-10-16 17:47:38 UTC
I am more and more suspecting my initial thought is correct - its environmental.

just by logging into the machine, I can see its telling me the network adapters and their IPs are configured in a special way to support multiple users running tests on these machines. The login messages say:

No IPs found. Try a shared IP: (Other people may be using these IPs.)
10.16.xxx.xxx
10.16.xxx.xxx
To use: export MYTESTIP_1=<ip address>
No IPs v6 found. Try a shared IP: (Other people may be using these IPs.)
...

So, if you are going to test on these machines, you have to make sure you are using only the IPs you have access to. I don't even think 127.0.0.1 is going to work!

Comment 4 John Mazzitelli 2013-10-16 19:33:00 UTC
this box appears to be locked down in a way I do not know how to work around.

Forget JON for a moment - just unzip a stock AS 7.1.1.Final, and run is like this:

./standalone.sh -b $MYTESTIP_1 -Djboss.bind.address.management=$MYTESTIP_1

where $MYTESTIP_1 is the 10.16.xxx.xxx IP address that we are told to use when we log in.

Now, try to connect to that via the JBoss AS 7.1.1.Final CLI:

$ ./jboss-cli.sh --connect --controller=${MYTESTIP_1}:9999
org.jboss.as.cli.CliInitializationException: Failed to connect to the controller
        at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:229)
        at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:207)
        at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.jboss.modules.Module.run(Module.java:260)
        at org.jboss.modules.Main.main(Main.java:291)
Caused by: org.jboss.as.cli.CommandLineException: The controller is not available at 10.16.xxx.xxx:9999
        at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:639)
        at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:613)
        at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:227)


Its the same error that JON gets when it tries to connect to the EAP instance during installation.

I don't know why this is - but suffice it to say, unless you work around the networking issues on this box, I don't think any JON testing can get done.

Comment 5 John Mazzitelli 2013-10-17 21:27:17 UTC
Looks like its an environment issue - perhaps Solaris Zone related... from IT:

====

Okay, that's convincing. I'm running the standalone.sh server and can see it
listening on port 9999

10.16.xxx.xxx.8080 *.* 0 0 128000 0 LISTEN
10.16.xxx.xxx.9999 *.* 0 0 128000 0 LISTEN
10.16.xxx.xxx.4447 *.* 0 0 128000 0 LISTEN
10.16.xxx.xxx.9990 *.* 0 0 128000 0 LISTEN

I'm able to connect to the server from my personal workstation but still can't
connect locally so there must be something going on with routing. I'm wondering
if it has to do with the fact that this is being tested on a Solaris zone (what
I believe to be a Solaris virtual machine hosted on Sparc hardware).

After further testing I've confirmed I can start standalone.sh and connect to
it locally on <HOSTNAME> which hosts zones dev32-01 and
dev32-02 so this does look like a zone issue. I'll have to look into whether we
can provide ssh access to the machine or not so you can run more tests since
it's technically a hypervisor, however.

Comment 6 Armine Hovsepyan 2013-10-18 12:56:57 UTC
the issue was environmental.
new environment is provided - dev32, rhq install works without issues on that environment.

closing the bug as not a bug.

Thanks for investigation John.