Bug 900707 (JBPAPP6-1405)

Any valid DNS name or IP breaks the parsing.
(If "foo" is used, and it resolves to a DNS name then it parses incorrectly.  It if does not resolve to a DNS name, it parses correctly).

Dennis, Could you please give a more detail guideline for reproduction of this issue, or to verify the "initial_hosts" value. As I tried modify domain.xml by adding : 

<protocol type="TCPPING">
  <property name="initial_hosts">localhost[7600]</property>
  <property name="prot_range">0</property>
  <property name="timeout">3000</property>
  <property name="num_initial_members">3</property>
</protocol>

but, that won't breaks the parsing or server start-up.

Read the description.  It's the property substitution that's broken.

Use a property substitution for initial_hosts with a default value containing any valid IP or DNS names
(and don't set the property, so it will try to use the default value).

<property name="initial_hosts">${some.property:localhost[7600]}</property>

In the JGroupsSubsystemXMLReader_1_1, I added some std-err output to see parsing property name and value from domain.xml. For the property substitution, no matter it can be resolve to a DNS name or not, it parses incorrectly such as:

If modify domain.xml by adding like:
<protocol type="TCPPING">
  property name="initial_hosts">${some.property:localhost[7600]}</property>
</protocol>

When start server, output info from my terminal are : 
[Host Controller] 14:50:08,485 INFO  [org.jboss.as.remoting] (MSC service thread 1-7) JBAS017100: Listening on 127.0.0.1:9999
[Host Controller] 14:50:08,874 ERROR [stderr] (Controller Boot Thread) property : initial_hosts
[Host Controller] 14:50:08,874 ERROR [stderr] (Controller Boot Thread) value : ${some.property:localhost[7600]}
[Host Controller] 14:50:08,874 ERROR [stderr] (Controller Boot Thread) properties : [("initial_hosts" => "${some.property:localhost[7600]}")]

On the other hand, if modify domain.xml by adding like:
<protocol type="TCPPING">
  <property name="initial_hosts">${some.property:foo[7600]}</property>
</protocol>

When start the server, output info from my terminal are :
[Host Controller] 14:51:19,668 INFO  [org.jboss.as.remoting] (MSC service thread 1-7) JBAS017100: Listening on 127.0.0.1:9999
[Host Controller] 14:51:20,046 ERROR [stderr] (Controller Boot Thread) property : initial_hosts
[Host Controller] 14:51:20,046 ERROR [stderr] (Controller Boot Thread) value : ${some.property:foo[7600]}
[Host Controller] 14:51:20,047 ERROR [stderr] (Controller Boot Thread) properties : [("initial_hosts" => "${some.property:foo[7600]}")]

With ${some.property:foo[7600]}
-------------------------------
Caused by: java.lang.Exception: Property assignment of initial_hosts in TCPPING with original property value foo[7600] and converted to null could not be assigned
        at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:1154) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.stack.Configurator.createLayer(Configurator.java:444) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.stack.Configurator.createProtocols(Configurator.java:393) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.stack.Configurator.setupProtocolStack(Configurator.java:88) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.stack.Configurator.setupProtocolStack(Configurator.java:55) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:465) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.JChannel.init(JChannel.java:795) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.JChannel.<init>(JChannel.java:166) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jboss.as.clustering.jgroups.MuxChannel.<init>(MuxChannel.java:37) [jboss-as-clustering-jgroups-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.clustering.jgroups.JChannelFactory.createChannel(JChannelFactory.java:73) [jboss-as-clustering-jgroups-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.clustering.jgroups.subsystem.ChannelService.start(ChannelService.java:40) [jboss-as-clustering-jgroups-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:65) [jboss-as-clustering-common-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        ... 4 more
Caused by: java.lang.Exception: Conversion of initial_hosts in TCPPING with original property value foo[7600] failed
        at org.jgroups.conf.PropertyHelper.getConvertedValue(PropertyHelper.java:85) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:1148) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        ... 15 more
Caused by: java.net.UnknownHostException: foo
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) [rt.jar:1.6.0_24]
        at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:850) [rt.jar:1.6.0_24]
        at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1201) [rt.jar:1.6.0_24]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1154) [rt.jar:1.6.0_24]
        at java.net.InetAddress.getAllByName(InetAddress.java:1084) [rt.jar:1.6.0_24]
        at java.net.InetAddress.getAllByName(InetAddress.java:1020) [rt.jar:1.6.0_24]
        at java.net.InetAddress.getByName(InetAddress.java:970) [rt.jar:1.6.0_24]
        at org.jgroups.stack.IpAddress.<init>(IpAddress.java:51) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]   
        at org.jgroups.util.Util.parseCommaDelimitedHosts(Util.java:2812) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.conf.PropertyConverters$InitialHosts.convert(PropertyConverters.java:63) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.conf.PropertyHelper.getConvertedValue(PropertyHelper.java:82) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        ... 16 more


With ${some.property:localhost[7600]}
-------------------------------------
Caused by: java.lang.Exception: String value could not be converted for method initial_hosts in TCPPING with default value ${some.property:localhost[7600]}.Exception is java.lang.Exception: Conversion of initial_hosts in TCPPING with original property value ${some.property:localhost[7600]} failed
        at org.jgroups.stack.Configurator.createInetAddressMap(Configurator.java:693) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.stack.Configurator.setupProtocolStack(Configurator.java:94) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.stack.Configurator.setupProtocolStack(Configurator.java:55) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:465) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.JChannel.init(JChannel.java:795) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.JChannel.<init>(JChannel.java:166) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jboss.as.clustering.jgroups.MuxChannel.<init>(MuxChannel.java:37) [jboss-as-clustering-jgroups-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.clustering.jgroups.JChannelFactory.createChannel(JChannelFactory.java:73) [jboss-as-clustering-jgroups-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.clustering.jgroups.subsystem.ChannelService.start(ChannelService.java:40) [jboss-as-clustering-jgroups-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:65) [jboss-as-clustering-common-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        ... 4 more
Caused by: java.lang.Exception: Conversion of initial_hosts in TCPPING with original property value ${some.property:localhost[7600]} failed
        at org.jgroups.conf.PropertyHelper.getConvertedValue(PropertyHelper.java:85) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.stack.Configurator.createInetAddressMap(Configurator.java:690) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        ... 13 more
Caused by: java.net.UnknownHostException: ${some.property:localhost
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) [rt.jar:1.6.0_24]
        at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:850) [rt.jar:1.6.0_24]
        at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1201) [rt.jar:1.6.0_24]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1154) [rt.jar:1.6.0_24]
        at java.net.InetAddress.getAllByName(InetAddress.java:1084) [rt.jar:1.6.0_24]
        at java.net.InetAddress.getAllByName(InetAddress.java:1020) [rt.jar:1.6.0_24]
        at java.net.InetAddress.getByName(InetAddress.java:970) [rt.jar:1.6.0_24]
        at org.jgroups.stack.IpAddress.<init>(IpAddress.java:51) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]   
        at org.jgroups.util.Util.parseCommaDelimitedHosts(Util.java:2812) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.conf.PropertyConverters$InitialHosts.convert(PropertyConverters.java:63) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        at org.jgroups.conf.PropertyHelper.getConvertedValue(PropertyHelper.java:82) [jgroups-3.0.9.Final-redhat-1.jar:3.0.9.Final-redhat-1]
        ... 14 more

I actually don't receive these exceptions from my console when I start up server with domain mode and deploy a simple WebApp, what setting or commands you executed to trigger that?

Dennis what are exact steps to reproduce this?

To reproduce:

Modify standalone/configuration/standalone-ha.xml:

Replace:
    <protocol type="MPING" socket-binding="jgroups-mping"/>

with:
                <protocol type="TCPPING">
                       <property name="initial_hosts">${some.property:localhost[7600]}</property>
                       <property name="port_range">0</property>
                       <property name="timeout">3000</property>
                       <property name="num_initial_members">3</property>
                </protocol>

Set the JGroups default-stack to "tcp".

bin/standalone.sh -c standalone-ha.xml

Error occurs during startup.

Great, thanks, seems like I made wangc poke around bad file.

thanks for that, Dennis

One little detail that escaped. To make AS7 start cluster, one needs to deploy distributable app: [https://github.com/jaysensharma/MiddlewareMagicDemos/blob/master/ClusterTest_WebApp/ClusterWebApp.war?raw=true]

Attachment: Added: standalone-ha.xml

More info from the support case.
It also occurs when substitution occurs.

Instead of ${some.property:localhost[7600]}, using ${some.property} and -Dsome.property=localhost[7600].
With the error message changed to: Conversion of initial_hosts in TCPPING with original property value ${some.property} failed
(where as using an invalid DNS name such as -Dsome.property=foo[7600] the error messages show it is trying to use foo).

I was able to dig into this a little deeper, and there are two issues:

1) the property substitution is not being done *at all* in EAP.
  This is part of a larger issue, where property substitution is missing for a whole lot of entries in standalone.xml/domain.xml.

2) since the property value is passed as is to JGroups, JGroups is running its own property substitution on it.
  There appears to be a bug related to this inside JGroups, where it tries to use the original value instead of the substituted value for specific properties containing IP addresses/host names.

I'll open a JGRP bug on #2.
Since JGroups does its own property substitution, which appears similar to what EAP would be doing, #1 probably isn't that important.

#2 in my previous comment is actually a bug in AS7, not JGroups.

clustering/jgroups/src/main/java/org/jboss/as/clustering/jgroups/JChannelFactory#createProtocol
creates a subclass of org.jgroups.conf.ProtocolConfiguration that overrides getOriginalProperties().
The new implementation is flawed, because it returns the original values before property substitution, 
where the parent version returns the values *after* property substition (but before any other changes).

So this is a bug that needs fixed in AS7.

I don't see the purpose of the overridden getOriginalProperties, as the only difference I see from the original is this bug.

As Dennis mentioned, one solution is resolve property substitution in jgroups' code, simply remove the flawed implementation can make it works fine. 

Attachment: Added: 0001-JBPAPP-9504-modify-flawed-implementation-of-getOrigi.patch

Another way is using PropertieValueResolver class in jboss.common.beans, so we can make the property substitution in AS, not with JGroups. 

Attachment: Added: 0001-JBPAPP-9504-use-the-PropertiesValueResolver-of-jboss.patch

This JIRA issue has a subject which indicates a general EAP6 problem and yet the examples presented only involve JGroups property substitution. Why wasn't this simply marked as a clustering issue and directed to the people who should be handling it? 
 

Here's the current status:

1. property substitution was introduced for JGroups protocol properties in AS7-4828 and made it into 7.2.0.Alpha branch but not 7.1 branch - it represents a change in the management API and should not go directly into EAP 6.0.x, as Brian noted 

2. The issue that Dennis observed relates to the use of one of the constructors in org.jgroups.conf.ProtocolConfiguration. I fixed the issue described in JBPAPP-9504 by using the alternative constructor, which means that a limited version of property substitution in JGroups protocol properties can now be used in 7.1. It's limited in the sense that substitutions can be coded, but rather than the AS management API processing them and handing resolved values to JGroups, the substitution expressions are passed directly to JGroups which then evaluates them before creating the channel.

However, I think introducing this into 7.1 is not desirable for two reasons:
2.1 this is a new feature for 7.1 and really doesn't belong in a bug fix release
2.2 this new feature is similar to management layer property substitution but different enough that it will cause confusion ; in particular, the expressions entered are not represented as expressions in the management model, and it breaks some of our existing CLI management operations, such as /subsystem=jgroups/stack=tcp:export-native-configuration

I'd rather make a release note saying that it is not available in 6.0.1. but will be in 7.2.0.

Thoughts?

I'll probably submit the pull request for the fix in any case, but need to chat with Bela for a few minutes tomorrow before doing so.

Richard

Thanks, Richard,

the pull request of https://issues.jboss.org/browse/AS7-4828 is not merged to master yet, I tested from this commit code, it made the default value problem in this issue resolved.

I do not see this as a feature issue.

The bug in itself must be fixed in a manner that will not cause an upwards compatibility problem with AS 7.2.

If that can't be done, then EAP 6.0.1 should throw an informative exception.

The issue was left unassigned because the correct default component "clustering" was not set, I expect.

I think an informative exception for branch 7.1 is the right way to go. System property substitution is not enabled by default for subsystem attribute values, and what we have here is a user who believes it is supported but in fact it is not (not yet, anyway). So an informative exception indicating this to the user that it's not enabled seems correct. 
 
 

Link: Added: This issue is related to AS7-5182

Release Notes Docs Status: Added: Needs More Info
Writer: Added: Darrin

Dennis,

Thanks for the feedback, will be updating the release note shortly.

bq. "There are a number of subsystems that don't have it enabled that should. JGroups is one of them."

I don't suppose there is a list of those subsystems available ? Or a JIRA for them ?

This should be working in 6.1 as https://issues.jboss.org/browse/AS7-4828 as per comment #21.

Agree with Dennis. I checked the EAP 6.x branch and the commit which fixes this issue is indeed present. So no further work need to be done here other than update the BZ status to reflect that.

Great. Let's close this as already fixed.