Bug 900801 (JBPAPP6-1420)

Summary: Adding modcluster via the CLI fails.
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Tom Fonteyne <tfonteyn>
Component: mod_clusterAssignee: Radoslav Husar <rhusar>
Status: CLOSED WONTFIX QA Contact: Michal Karm Babacek <mbabacek>
Severity: high Docs Contact:
Priority: high    
Version: 6.0.0CC: akostadi, brian.stansberry, jclere, mbabacek, myarboro, paul.ferraro, rhatlapa, rhusar, tcerar, tfonteyn, wfink
Target Milestone: ER6   
Target Release: EAP 6.1.0   
Hardware: Unspecified   
OS: Unspecified   
URL: http://jira.jboss.org/jira/browse/JBPAPP6-1420
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
There exists an issue in JBoss EAP 6 where the `mod_cluster` subsystem cannot be added by a single command via the Management CLI, nor can it be added by a sequence of commands that are executed individually. The `/subsystem=modcluster:add()` command requires that a connector be added as part of the same set of commands, however the `/subsystem=modcluster:add()` command itself does not allow connectors to be configured. Attempting to add the `mod_cluster` subsystem via the Management CLI fails with an error: `JBAS014746: connector may not be null`. To workaround this issue, execute the `add()` and connector configuration commands in batch mode. For example: ---- [standalone@localhost:9999 /] batch [standalone@localhost:9999 / #] /subsystem=modcluster:add [standalone@localhost:9999 / #] /subsystem=modcluster/mod-cluster-config=configuration:add(connector=ajp,advertise-socket=modcluster) [standalone@localhost:9999 / #] run-batch ---- This issue will be fixed in a future release.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-16 20:30:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tom Fonteyne 2012-08-21 09:46:04 UTC
Help Desk Ticket Reference: https://c.na7.visual.force.com/apex/Case_View?id=500A000000AtAfhIAF
Steps to Reproduce: Take a standard standalone install.

Add to extensions:
/extension=org.jboss.as.modcluster:add()

now add the subsystem:
/subsystem=modcluster:add()                
{
    "outcome" => "failed",
    "failure-description" => "JBAS014746: connector may not be null",
    "rolled-back" => true
}

but the add() operation does not allow any parameters.
Workaround: Workaround Exists
Workaround Description: /:composite(steps=[ {"operation" => "add", "address" => [ ("subsystem" => "modcluster") ] },{ "operation" => "add", "address" => [ ("subsystem" => "modcluster"), ("mod-cluster-config" => "configuration") ], "connector" => "ajp", "balancer" => "my", "advertise" => "false", "proxy-list" => "zen" } ])

Note that the AJP connector needs to exist, if not, create it with something like this:
/subsystem=web/connector=ajp:add(name=ajp,protocol=ajp,scheme=ajp,socket-binding=ajp)
project_key: JBPAPP6

Adding modcluster via the CLI fails.

Comment 1 Tom Fonteyne 2012-08-21 09:47:30 UTC
Help Desk Ticket Reference: Added: https://c.na7.visual.force.com/apex/Case_View?id=500A000000AtAfhIAF
Forum Reference: Added: https://community.jboss.org/wiki/AddingANewExtensionSubsystemViaTheCLI


Comment 2 Tom Fonteyne 2012-08-21 09:48:09 UTC
Link: Added: This issue Cloned to JBPAPP-9717


Comment 3 Anne-Louise Tangring 2012-11-13 20:57:57 UTC
Docs QE Status: Removed: NEW 


Comment 4 Brian Stansberry 2013-03-06 19:35:26 UTC
There shouldn't be any expectation of any change in this in 6.1 beyond any improvements made in time for the 6.1.0 DR4/ER1 release.

Comment 8 Jean-frederic Clere 2013-04-09 06:42:59 UTC
first add modcluster and then the configuration.

Comment 9 Michal Karm Babacek 2013-04-11 09:20:54 UTC
Guys, no. It is still broken. (EAP6.1.0.ER4)

1) connector is ajp implicitly idea: Does not work.
/subsystem=modcluster:add()                
{
    "outcome" => "failed",
    "failure-description" => "JBAS014746: connector may not be null",
    "rolled-back" => true
}

2) let's add it in disabled state idea: Does not work.
/subsystem=modcluster:add(enable=false)
Operation 'add' does not expect any property.


3) let's add it with connector=ajp idea: Does not work.
/subsystem=modcluster:add(connector=ajp)
Operation 'add' does not expect any property.

4) with the "configuration" level, similar to 3), Does not work.
/subsystem=modcluster:add(configuration=[{advertise-socket=modcluster,connector=ajp}])
Operation 'add' does not expect any property.

5) composite operation idea: Does not work.

[standalone@localhost:9999 / #] list-batch
#1 /:composite(steps=[{"operation" => "add", "address" => [ ("subsystem" => "modcluster") ] },{ "operation" => "add", "address" => [ ("subsystem" => "modcluster"), ("mod-cluster-config" => "configuration") ], "connector" => "ajp"}])

[standalone@localhost:9999 / #] run-batch
{"JBAS014653: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-1" => {"JBAS014653: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-1" => {"JBAS014671: Failed services" => {"jboss.mod-cluster" => "org.jboss.msc.service.StartException in service jboss.mod-cluster: java.lang.IllegalStateException
    Caused by: java.lang.IllegalStateException"}}}}}}

In log further:
10:59:56,694 INFO  [org.jboss.as.modcluster] (management-handler-thread - 4) JBAS011704: Mod_cluster uses default load balancer provider
10:59:56,698 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 48) MSC000001: Failed to start service jboss.mod-cluster: org.jboss.msc.service.StartException in service jboss.mod-cluster: java.lang.IllegalStateException
	at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87) [jboss-as-clustering-common-7.2.0.Final-redhat-4.jar:7.2.0.Final-redhat-4]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_15]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_15]
	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
Caused by: java.lang.IllegalStateException
	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at org.jboss.as.modcluster.ModClusterService.start(ModClusterService.java:94)
	at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82) [jboss-as-clustering-common-7.2.0.Final-redhat-4.jar:7.2.0.Final-redhat-4]
	... 4 more


Yes, I have /extension=org.jboss.as.modcluster:add() in place :-)
Yes, I have /subsystem=web/connector=ajp:add(name=ajp,protocol=ajp,scheme=ajp,socket-binding=ajp) in place.

Conclusion: I am not able to add modcluster subsystem via CLI while having standalone.xml configuration in place.

Comment 10 Tom Fonteyne 2013-04-11 13:47:13 UTC
There seems to be a regression. The composite command as given as workaround works fine in EAP 6.0.1

So I agree with Michal. The "normal" way of adding the subsystem still does not work (just like in 6.0.x) and the "composite" way that used to work is broken now.

Comment 11 Michal Karm Babacek 2013-04-16 12:25:09 UTC
(In reply to comment #10)
> There seems to be a regression. The composite command as given as workaround
> works fine in EAP 6.0.1
> 
> So I agree with Michal. The "normal" way of adding the subsystem still does
> not work (just like in 6.0.x) and the "composite" way that used to work is
> broken now.

COMPOSITE OPERATION
Hmm, no, I was too quick in my judgement. The undermentioned composite operation actually _works_ both in 6.0.1 and 6.1.0. Obviously, the composite operation from my previous comment [1] is wrong as you might have noticed. Sorry for confusion.

Add extension
/extension=org.jboss.as.modcluster:add()

Add ajp connector
/subsystem=web/connector=ajp:add(name=ajp,protocol=ajp,scheme=ajp,socket-binding=ajp)

Composite operation for mod_cluster
/:composite(steps=[ {"operation" => "add", "address" => [ ("subsystem" => "modcluster") ] },{ "operation" => "add", "address" => [ ("subsystem" => "modcluster"), ("mod-cluster-config" => "configuration") ], "connector" => "ajp", "balancer" => "my", "advertise" => "false", "proxy-list" => "127.0.0.1:6666" } ])

[1] https://bugzilla.redhat.com/show_bug.cgi?id=900801#c9

Comment 14 Tomaz Cerar 2013-04-18 15:02:47 UTC
What should be accepted solution to this?
Given that composite operation (cli batch) works as expected and verified?

In EAP 6.0.x modcluster:add() operation also did not expect or take any attributes:

output of operation definition in EAP 6.0.0.GA
[standalone@localhost:9999 subsystem=modcluster] :read-operation-description(name=add)
{
    "outcome" => "success",
    "result" => {
        "operation-name" => "add",
        "description" => "Operation adding the modcluster subsystem.",
        "request-properties" => {},
        "reply-properties" => {},
        "read-only" => false
    }
}

Said this, I see no need to add new feature such as adding convenience attribute to subsystem=modcluster:add this late in development stage.
I could be nice RFE for 6.2 but absolutely not for 6.1 as it would require lots of additional work (model version upgrade, transformers, additional testing)

Comment 15 Brian Stansberry 2013-04-18 15:31:39 UTC
Tomaz: agreed. This is what I meant in my comment on 2013-03-06.

Comment 18 Michal Karm Babacek 2013-04-18 18:57:21 UTC
As I stated in [1] on 2013-04-16, the composite operation mentioned _in that comment_ works. My previous error was in using a malformed command.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=900801#c11

Comment 19 Brian Stansberry 2013-04-18 19:05:59 UTC
Thanks. To clarify my last clarification .... ;)

If the composite operation mentioned in Michal's comment of Apri 16 works, IMHO there is no blocker, and no remaining "Bug", and at most an RFE for a later release.

Comment 20 Michal Karm Babacek 2013-04-19 16:03:18 UTC
Documentation
-------------

If one wants to use advertise=true and no proxy-list, a proper socket-binding must be added. Here is the whole batch op executed from a command line:

./jboss-cli.sh --connect --commands="
/extension=org.jboss.as.modcluster:add(),
/subsystem=web/connector=ajp:add(name=ajp,protocol=ajp,scheme=ajp,socket-binding=ajp),
batch,
/:composite(steps=[ 
  {\"operation\" => \"add\", \"address\" => [ (\"subsystem\" => \"modcluster\") ] },
  { \"operation\" => \"add\", \"address\" => [ (\"subsystem\" => \"modcluster\"), (\"mod-cluster-config\" => \"configuration\") ], \"connector\" => \"ajp\", \"advertise-socket\" => \"modcluster\" },
  {\"operation\" => \"add\", \"address\" => [(\"socket-binding-group\" => \"standard-sockets\"), (\"socket-binding\" => \"modcluster\")], \"port\" => 0, \"multicast-address\" => \"224.0.1.105\", \"multicast-port\" => \"23364\"}
]),
run-batch"

(Note socket-binding and advertise-socket.)

Comment 21 Jean-frederic Clere 2013-04-23 12:51:22 UTC
According Comment #19 and Comment #20 that is not a blocker.

Comment 22 John Doyle 2013-04-23 12:54:14 UTC
I'm in favor of removing blocker status.

Comment 24 Aleksandar Kostadinov 2013-05-03 07:57:35 UTC
I am running EAP 6.0.1 ER5 in *domain* mode. And there seems to be no way to add mod_cluster through cli while there are running servers.

Foremost composite operation does not seem to work at all. Any composite operation I try to execute results in:
> Operation 'composite' does not expect any property.

This command has worked with EAP 6.0.1 but is *not* working now. This is a *regression*:

> /:composite(steps=[ {"operation" => "add", "address" => [ ("profile" => "default"), ("subsystem" => "modcluster") ] },{ "operation" => "add", "address" => [ ("profile" => "default"), ("subsystem" => "modcluster"), ("mod-cluster-config" => "configuration") ], "advertise" => "false", "proxy-list" => "${jboss.modcluster.proxyList}", "connector" => "ajp"}, { "operation" => "add", "address" => [ ("profile" => "default"), ("subsystem" => "modcluster"), ("mod-cluster-config" => "configuration"), ("dynamic-load-provider" => "configuration") ]}, { "operation" => "add", "address" => [ ("profile" => "default"), ("subsystem" => "modcluster"), ("mod-cluster-config" => "configuration"), ("dynamic-load-provider" => "configuration"), ("load-metric" => "busyness")], "type" => "busyness"} ])
> Operation 'composite' does not expect any property.

Trying with normal commands I'm getting the famous "connector may not be null"

> /profile=default/subsystem=modcluster:add
> {
>     "outcome" => "failed",
>     "result" => undefined,
>     "failure-description" => "JBAS010839: Operation failed or was rolled back on all servers.",
>     "rolled-back" => true,
>     "server-groups" => {"main-server-group" => {"host" => {"ip-10-0-254-48" => {
>         "server-one" => {"response" => {
>             "outcome" => "failed",
>             "result" => undefined,
>             "failure-description" => "JBAS014746: connector may not be null",
>             "rolled-back" => true
>         }},
>         "server-two" => {"response" => {
>             "outcome" => "failed",
>             "result" => undefined,
>             "failure-description" => "JBAS014746: connector may not be null",
>             "rolled-back" => true
>         }}
>     }}}}
> }

The same command works only if server-one and server-two both are stopped. The composite command does not start working though.

Comment 25 Michal Karm Babacek 2013-05-03 09:09:25 UTC
(In reply to comment #24)
> I am running EAP 6.0.1 ER5 in *domain* mode. And there seems to be no way to
> add mod_cluster through cli while there are running servers.

Did you mean EAP 6.1.0 ER5? Regarding my Comment 20, I tested the standalone mode.

Comment 26 Aleksandar Kostadinov 2013-05-03 10:05:32 UTC
Michal, that's right, I meant 6.1.0 ER5

Comment 27 Brian Stansberry 2013-05-03 16:34:28 UTC
Aleksandar, what happens when you use 

batch
...
run batch

instead of building the composite yourself?

Comment 28 Radoslav Husar 2013-05-03 17:34:54 UTC
I cannot reproduce this issue. I have created 2 servers using default configuration, where I removed mod_cluster subsystem. I let the servers autostart. I start the domain.

I run the following batch:

> [domain@localhost:9999 /] batch 
> [domain@localhost:9999 / #] /profile=default/subsystem=modcluster:add
> #1 /profile=default/subsystem=modcluster:add
> [domain@localhost:9999 / #] /profile=default/subsystem=modcluster/mod-cluster-config=configuration:add(advertise=false, proxy-list="localhost:6666", connector=http)
> #2 /profile=default/subsystem=modcluster/mod-cluster-config=configuration:add(advertise=false, proxy-list="localhost:6666", connector=http)
> [domain@localhost:9999 / #] /profile=default/subsystem=modcluster/mod-cluster-config=configuration/dynamic-load-provider=configuration:add
> #3 /profile=default/subsystem=modcluster/mod-cluster-config=configuration/dynamic-load-provider=configuration:add
> [domain@localhost:9999 / #] /profile=default/subsystem=modcluster/mod-cluster-config=configuration/dynamic-load-provider=configuration/load-metric=busyness:add(type=busyness)
> #4 /profile=default/subsystem=modcluster/mod-cluster-config=configuration/dynamic-load-provider=configuration/load-metric=busyness:add(type=busyness)
> [domain@localhost:9999 / #] run-batch 
> The batch executed successfully


I get correct logging (from server1) which I am expecting:

> 19:29:48,306 DEBUG [org.jboss.as.controller.management-operation] (host-controller-connection-threads - 1) Entered VERIFY stage; waiting for service container to settle
> 19:29:48,306 DEBUG [org.jboss.as.modcluster] (ServerService Thread Pool -- 48) Starting Mod_cluster Extension
> 19:29:48,327 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 48) MODCLUSTER000001: Initializing mod_cluster ${project.version}
> 19:29:48,331 DEBUG [org.jboss.modcluster] (ServerService Thread Pool -- 48) MODCLUSTER000003: Received server start event
> 19:29:48,334 ERROR [org.jboss.modcluster] (ServerService Thread Pool -- 48) MODCLUSTER000043: Failed to send INFO to localhost/127.0.0.1:6666: java.net.ConnectException: Connection refused

@Alex, what should I do differently to hit an issue?

Comment 29 Radoslav Husar 2013-05-03 17:40:16 UTC
@Alex, I see your problem now. You seem to be using the stock profile to test this, however:

You are specifying the "ajp" connector as a connector for mod_cluster. Note that AJP connector has been *removed* from the default profile in EAP 6.1.0, whereas it was present in EAP 6.0.1. The default profile now only specifies "http" connector.

Comment 30 Brian Stansberry 2013-05-03 18:05:21 UTC
@Alex, looking into this, this "Operation 'composite' does not expect any property issue" has nothing to do with the mod_cluster subsystem. The things Rado has pointed out may influence what results you would see when trying to isolate this issue, but it's a general problem with manually entered composite operations.

I'll file a separate issue.

Comment 31 Brian Stansberry 2013-05-03 18:17:27 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=959545 is for the composite operation issue.

Comment 33 Aleksandar Kostadinov 2013-05-06 07:02:17 UTC
Now I see. My problem was not the AJP connector, because I add it prior running the commands.
First (that might be obvious to others) but the modcluster :add command fails when run outside a batch. It makes sense so it is not a bug.

I now see though why my batch command attempts failed but I didn't pay enough attention to it before. It seems that if I paste a couple of commands to the console, it ignores the new lines so it considers everything as one single command which is incorrect.

So if I paste one line at a time and hit ENTER after each one, everything works fine. If I paste a couple of lines, then if fails with some error for incorrect command. This was not the case in the past. I'll file a separate bug for it because it is unrelated.

Comment 35 Jean-frederic Clere 2013-05-22 11:47:32 UTC
Cause: 

The modcluster subsystem can't be added without specifying a connector and the add of modcluster subsystem doesn't accept any parameter.

Consequence: 

you can't add the modcluster subsystem via the cli.

Workaround (if any): 

you need to use a batch something like:
batch
/subsystem=modcluster:add
/subsystem=modcluster/mod-cluster-config=configuration:add(connector=ajp,advertise-socket=modcluster)
run-batch

Result: 
.

Comment 36 Brian Stansberry 2013-08-05 16:47:57 UTC
The proposed docs text is ok. A more exact first paragraph would be:

"There exists an issue in JBoss EAP 6 where the `mod_cluster` subsystem cannot be added via a single simple command in the Management CLI, nor can it be added via a simple sequence of non-atomic individual commands. The `/subsystem=modcluster:add()` command requires that a connector be added as part of the same set of commands, however the `/subsystem=modcluster:add()` command does not itself allow configuring connectors."

The above is pretty wordy and legalistic though. :)

Comment 37 Lucas Costi 2013-08-06 00:47:28 UTC
Thanks for the feedback Brian. I have incorporated your suggestion into the docs text with some minor refinements. :)

Let us know if anything else needs needs to be changed.