The agent is starting fine. From the logs:

-----
11:23:55,756 INFO  [org.hawkular.agent.monitor.service.AgentCoreEngine] (Hawkular Java Agent Start Thread) HAWKMONITOR010000: Starting Hawkular Agent
...
11:23:56,679 INFO  [org.hawkular.agent.monitor.cmd.FeedCommProcessor] (Hawkular Java Agent Start Thread) HAWKMONITOR010039: The command-gateway URL is [ws://docker-haw-services.bc.jonqe.lab.eng.bos.redhat.com:8080/hawkular/command-gateway/feed/3013b67aa3dd]
11:23:56,724 INFO  [org.hawkular.agent.monitor.cmd.FeedCommProcessor] (OkHttp http://docker-haw-services.bc.jonqe.lab.eng.bos.redhat.com:8080/...) HAWKMONITOR010034: Opened feed WebSocket connection to endpoint [ws://docker-haw-services.bc.jonqe.lab.eng.bos.redhat.com:8080/hawkular/command-gateway/feed/3013b67aa3dd]
11:23:56,769 INFO  [org.hawkular.agent.monitor.scheduler.SchedulerService] (Hawkular Java Agent Start Thread) HAWKMONITOR010012: Starting scheduler
11:23:56,799 INFO  [org.hawkular.agent.monitor.protocol.ProtocolServices] (Hawkular Java Agent Start Thread) HAWKMONITOR010019: Managed server [Remote DMR EAP6] is disabled. It will not be monitored.
-----

Looks like perhaps its yaml config file doesn't have any managed servers enabled? Is it using the correct config file?

Can you attach the config file found at /opt/hawkular/etc/hawkular-javaagent-config.yaml to this BZ?

Created attachment 1281556 [details]
hawkular javaagent config

Attaching hawkular-javaagent-config.yaml from latest eap6 container.

I missed this earlier. In the log output:

11:23:57,215 INFO  [org.hawkular.agent.monitor.protocol.EndpointService] (Hawkular Java Agent Start Thread) Endpoint [DMREndpointService[Endpoint[Local DMR]:[http-remoting://127.0.0.1:9990]]] is not ready yet - will wait

So this means the "Local DMR" managed server definition is enabled and the agent will begin to monitor it, but it first is going to wait for the "wait-for" resources to be available.

The "wait-for" resource defined is the root resource ("/") - from the config file:

====
managed-servers:
  local-dmr:
    name: Local DMR
...
    wait-for:
    - name: /
====

This tells me that most likely the management interface for this EAP6 server is not reachable at http-remoting://127.0.0.1:9990.

Can you confirm you can go over that IP and port to talk to the management interface. Try this - in the EAP6 bin directory, run the jboss-cli.sh with this command line:

./jboss-cli.sh --connect '--command=/socket-binding-group=standard-sockets/socket-binding=management-http/:read-resource(include-runtime=true,attributes-only=true)'

If you get back results here, I think the reason is because the local-dmr default protocol of "http-remoting" is not supported by EAP6.

If that is the case, I think the easiest way to work around this is to just configure the "Local DMR" as really a remote-dmr (in the javaagent, local-dmr is just a remote-dmr with some default settings, so switching to using remote-dmr is really not going to be much different).

So change the local-dmr section in the config so it is disabled and change its name (because we want to use "Local DMR" as our name for the remote-dmr):

====
managed-servers:
  local-dmr:
    name: DISABLED Local DMR
    enabled: false
====

And now enable the remote-dmr entry and set its name to "Local DMR":

====
  remote-dmr:
  - name: Remote DMR EAP6
    enabled: false
    protocol: remote
    host: 127.0.0.1
    port: 9999
    username: adminUser
    password: adminPass
    ... and the rest ...
====

Notice the "protocol" is set to "remote". I do not know offhand if you have to specify username and password (if you are connecting to the server from the same machine where the client is, you might not need to do this. But if you do, just add a admin user).

If the problem is what I think it is, and we want to fix this in code rather than in the config, we'll have to change the default protocol to "remoting" if we detect the server is an EAP6 server (if its EAP7, then http-remoting can be used).

For the record, I think this is the code that is important:

https://github.com/hawkular/hawkular-agent/blob/master/hawkular-javaagent/src/main/java/org/hawkular/agent/javaagent/config/ConfigConverter.java#L302

I confirmed the problem is as I thought. I performed the workaround described in my earlier comment (local-dmr is disabled, remote-dmr is enabled and named "Local DMR") and I see the agent starting to monitor.

So we can either change the image's config (which is the fastest way to fix this) or we change the agent code to be smarter about its default (if running on EAP6, the default protocol is "remote" - anything else, default protocol is "http-remoting"). That would require new releases to be made.

Tell me what way you want to go.

Given the time constraints I suggest we change the image configuration.

(In reply to John Doyle from comment #9)
> Given the time constraints I suggest we change the image configuration.

Then the way i would do it is take the current yaml, remove the "local-dmr" entry entirely (since it can't work on the older EAP6 protocol) and change the name of the remote-dmr entry to "Local DMR" and set its "enabled" setting to "true".

Everything else can be left alone and it should work.

I have a PR that can fix this by setting a sysprop. When you start EAP6, you pass in "-Dhawkular.local.dmr.protocol=remote" and it should work.

The PR isn't merged yet, nothing has been released yet, but the code is here in case we want to use it.

See PR: https://github.com/hawkular/hawkular-agent/pull/344

That agent PR passed all tests, Jay S. reviewed and merged it into master.

So if need be, we have the code ready. It would just require the EAP6 docker image to pass in the command line option "-Dhawkular.local.dmr.protocol=remote" when starting the server that has the Java Agent attached.

The agent with this code has not been released as of the time of this comment being posted.

Let's try to get this in before the CF release.

Let's try to get this in before the CF release.

If we want to use the new agent code rather than change the yaml config, the agent has been released - 1.0.0.CR5.

Aside from picking up the new agent version, the docker image needs to pass the following cmd line argument to the EAP6 server:

   -Dhawkular.local.dmr.protocol=remote

@Matt @Mazz, can you please agree ASAP on what's necessary and then ask Rob to update the EAP6 image ?

(In reply to Thomas Heute from comment #18)
> @Matt @Mazz, can you please agree ASAP on what's necessary and then ask Rob
> to update the EAP6 image ?

There was a conversation via email and on the chat room with Rob on this. I believe Rob was switching to using remote-dmr in the EAP6 image:

(12:50:08 PM) rcernich: mazz: would it be acceptable to just change the hawkular config to activate remote instead of adding the system property?
(12:50:31 PM) mazz: rcernich: yes, that is the other alternative solution
(12:50:38 PM) mazz: that is mentioned in the BZ
(12:50:52 PM) rcernich: mazz: that's easier, since we already have different configs for eap 6 and 7
(12:51:14 PM) mazz: in that case, you would set "enabled: false" in the local-dmr setting in the yaml config
(12:51:22 PM) rcernich: yep
(12:51:36 PM) mazz: and in the remote-dmr config, "enabled: true" and set the name to "Local DMR" because that names has to match that or MiQ console will break IIRC
(12:52:05 PM) rcernich: mazz: ahh, ok
...
(12:54:37 PM) mazz: rcernich: I forgot the most important thing when changing the config :}
(12:54:47 PM) mazz: the whole point to moving to remote-dmr was so you can tell it the protocol
(12:55:02 PM) mazz: in the remote-dmr you set the "protocol: remote" setting
(12:55:38 PM) rcernich: mazz: that's what's in there now
(12:55:43 PM) mazz: ok, good.
(12:55:56 PM) rcernich: mazz: it's using the config from the javaagent examples
(12:55:58 PM) rcernich: lemme find it
(12:58:24 PM) rcernich: mazz: sorry, it uses this one: https://github.com/hawkular/hawkular-agent/blob/master/eap6-support/hawkular-javaagent-wildfly-feature-pack-eap6/src/main/resources/featurepack/content/standalone/configuration/hawkular-javaagent-config.yaml
...
(01:13:11 PM) rcernich: mazz: ok, so given the config above, i need to set local-dmr.enabled=false, remote-dmr.enabled=true and remote-dmr.name=Local DMR, correct?
(01:15:19 PM) mmahoney|afk is now known as mmahoney
(01:15:52 PM) mazz: rcernich: correct

For the record, this is why we need to do this - http-remoting procotol causes the following exception when talking to EAP6:

(01:03:57 PM) mazz: org.hawkular.dmr.api.DmrApiException: java.io.IOException: java.net.ConnectException: WFLYPRT0053: Could not connect to http-remoting://127.0.0.1:9990. The connection failed
(01:05:07 PM) mazz: org.xnio.http.RedirectException: XNIO000816: Redirect encountered establishing connection
(01:05:18 PM) mazz: that's the cause of the cause
(01:07:57 PM) mazz: new org.jboss.as.controller.client.ModelControllerClientConfiguration.Builder().setProtocol("http-remoting")....build()
(01:08:20 PM) mazz: that gets me the config which is passed to org.jboss.as.controller.client.ModelControllerClient.Factory.create(theconfig)

@Mazz, so just confirming that with the fix that was implemented, we need to document that  when you start the EAP6 container image, you need to pass in the following command line argument:

-Dhawkular.local.dmr.protocol=remote

(In reply to jstickle from comment #21)
> @Mazz, so just confirming that with the fix that was implemented, we need to
> document that  when you start the EAP6 container image, you need to pass in
> the following command line argument:
> 
> -Dhawkular.local.dmr.protocol=remote

If the solution to this BZ is going to be upgrading to the latest agent and to use this new feature, then that sysprop needs to be passed in, yes. But usually the user doesn't have to worry about it - the docker image that we produce should have that parameter passed in for you. You run the image and it "should just work" without the user having to pass in this sys prop manually.

Verified on revision: jboss-eap-6/eap64-openshift:1.5-7

I confirm that EAP6 server successfully connects to Middleware Manager is is shown in CFME as immutable.