Bug 1272528 - New Storage Node property file wrong parameters
New Storage Node property file wrong parameters
Status: CLOSED NOTABUG
Product: JBoss Operations Network
Classification: JBoss
Component: Storage Node (Show other bugs)
JON 3.3.4
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Michael Burman
Mike Foley
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-16 12:03 EDT by Vojta Prusa
Modified: 2016-06-20 05:37 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
RHEL 7.1 Server and remote agent registered, default location of storage
Last Closed: 2015-10-29 10:45:39 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Vojta Prusa 2015-10-16 12:03:03 EDT
Description of problem:
Some parameters in new (ER2 patch) property file for cassandra node (./jon-server-3.3.0.GA/rhq-storage/conf/cassandra-jvm.properties.new) shows errors on JON server start (./current-jon/bin/rhqctl start)


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Update JON server (ER2 patch)
2. Update new cassandra node properties: cp ./jon-server-3.3.0.GA/rhq-storage/conf/cassandra-jvm.properties.new ./jon-server-3.3.0.GA/rhq-storage/conf/cassandra-jvm.properties
4. Start JON server: ./jon-server-3.3.0.GA/bin/rhqctl start

Actual results:


Expected results:
Clean start storage node Up

Additional info:
Parameters:
(New - from ./jon-server-3.3.0.GA/rhq-storage/conf/cassandra-jvm.properties.new)
(Should be: ./jon-server-3.3.0.GA/rhq-storage/conf/cassandra-jvm.properties)
(Info: description + errors)

New: heap_dump_dir=
Should be: heap_dump_dir=/home/hudson/jon-server-3.3.0.GA/rhq-storage/bin
Info: Not 100% sure if this one hadnt changed while testing, but needs to be set

New: thread_stack_size="-Xss180k"
Should be: thread_stack_size="-Xss256k"
Info: New value throws error:
[hudson@vprusa-jon-er1-s1 ~]$ ./current-jon/bin/rhqctl start
11:12:01,240 INFO  [org.jboss.modules] JBoss Modules version 1.3.7.Final-redhat-1
The stack size specified is too small, Specify at least 228k
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Trying to start the RHQ Server...
RHQ Server                     (pid 6282   ) is starting
Starting RHQ Agent...
RHQ Agent (pid 6607 ) is ✔running

New: java_agent=
Should be: java_agent="-javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar"
Info: Empty parameter value throws error:
[hudson@vprusa-jon-er1-s1 ~]$ ./current-jon/bin/rhqctl start
11:16:19,474 INFO  [org.jboss.modules] JBoss Modules version 1.3.7.Final-redhat-1
 INFO 11:16:19,863 Logging initialized
 INFO 11:16:19,951 JVM vendor/version: OpenJDK 64-Bit Server VM/1.7.0_75
 INFO 11:16:19,952 Heap size: 261750784/987365376
 INFO 11:16:19,952 Classpath: ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-1.2.18-jboss-2.jar:./../lib/apache-cassandra-clientutil-1.2.18-jboss-2.jar:./../lib/apache-cassandra-thrift-1.2.18-jboss-2.jar:./../lib/avro-1.4.0-fixes.jar:./../lib/avro-1.4.0-sources-fixes.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.7.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.2.0.jar:./../lib/netty-3.6.6.Final.jar:./../lib/netty-3.9.1.Final.jar:./../lib/rhq-cassandra-auth-4.12.0.JON330GA.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.6.jar:./../lib/snappy-java-1.0.4.1-rhq-p1.jar:./../lib/snaptree-0.1.jar
 INFO 11:16:19,954 JNA not found. Native methods will be disabled.
 WARN 11:16:19,956 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
 INFO 11:16:20,079 Loading settings from file:/home/hudson/jon-server-3.3.0.GA/rhq-storage/conf/cassandra.yaml
Trying to start the RHQ Server...
RHQ Server                     (pid 7816   ) is starting
Starting RHQ Agent...
RHQ Agent (pid 8138 ) is ✔running

Also:
[hudson@vprusa-jon-er1-s1 ~]$ ./current-jon/bin/rhqctl status
11:17:05,287 INFO  [org.jboss.modules] JBoss Modules version 1.3.7.Final-redhat-1
RHQ Storage Node               (pid 7583   ) is ✔running
RHQ Server                     (pid 7816   ) is ✔running
JBossAS Java VM child process  (pid 7935   ) is ✔running
RHQ Agent                      (pid 8138   ) is ✔running 

!But RHQ Storgae Node is Down in GUI

New: JMX_OPTS="-Dcassandra.jmx.local.port=${jmx_port}"
Should be: JMX_OPTS="-Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote.port=${jmx_port} -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
Info: Otherwise resource is Down in GUI and:

[hudson@vprusa-jon-er1-s1 ~]$ ./current-jon/bin/rhqctl start
11:20:15,258 INFO  [org.jboss.modules] JBoss Modules version 1.3.7.Final-redhat-1
 INFO 11:20:15,676 Logging initialized
 INFO 11:20:15,765 JVM vendor/version: OpenJDK 64-Bit Server VM/1.7.0_75
 INFO 11:20:15,765 Heap size: 261750784/987365376
 INFO 11:20:15,765 Classpath: ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-1.2.18-jboss-2.jar:./../lib/apache-cassandra-clientutil-1.2.18-jboss-2.jar:./../lib/apache-cassandra-thrift-1.2.18-jboss-2.jar:./../lib/avro-1.4.0-fixes.jar:./../lib/avro-1.4.0-sources-fixes.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.7.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.2.0.jar:./../lib/netty-3.6.6.Final.jar:./../lib/netty-3.9.1.Final.jar:./../lib/rhq-cassandra-auth-4.12.0.JON330GA.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.6.jar:./../lib/snappy-java-1.0.4.1-rhq-p1.jar:./../lib/snaptree-0.1.jar:./../lib/jamm-0.2.5.jar
 INFO 11:20:15,768 JNA not found. Native methods will be disabled.
 WARN 11:20:15,771 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
 INFO 11:20:15,899 Loading settings from file:/home/hudson/jon-server-3.3.0.GA/rhq-storage/conf/cassandra.yaml

...MORE INFO: Opening files...

INFO 11:20:17,883 Initializing rhq.metrics_idx
Trying to start the RHQ Server...
RHQ Server                     (pid 9868   ) is starting
Starting RHQ Agent...
RHQ Agent (pid 10186 ) is ✔running

[hudson@vprusa-jon-er1-s1 ~]$ ./current-jon/bin/rhqctl status
11:23:24,129 INFO  [org.jboss.modules] JBoss Modules version 1.3.7.Final-redhat-1
RHQ Storage Node               (pid 9634   ) is ✔running
RHQ Server                     (pid 9868   ) is ✔running
JBossAS Java VM child process  (pid 9987   ) is ✔running
RHQ Agent                      (pid 10186  ) is ✔running 


Relates to: https://bugzilla.redhat.com/show_bug.cgi?id=1272239
Comment 1 Sunil Kondkar 2015-10-19 12:00:55 EDT
Additional Info:
Facing the same issue when trying to verify bug#1230411. After applying the patch to already installed JBoss ON 3.3GA, and merging the 'cassandra-jvm.properties.new' with 'cassandra-jvm.properties', GUI shows the storage node availability as down.
Comment 2 John Sanda 2015-10-19 13:23:27 EDT
Can you share your new/updated cassandra-jvm.properties? In particular I am interested in the JMX_OPTS property. Take a look at https://bugzilla.redhat.com/show_bug.cgi?id=1229978#c4. The JMX_OPTS property should be changed to,

JMX_OPTS="-Djava.rmi.server.hostname=localhost"
Comment 3 Filip Brychta 2015-10-20 11:39:16 EDT
John, could you clarify if JMX_OPTS from previous comment are correct? Because looking at bz1229978 comment 4, cassandra-jvm.properties.new contains JMX_OPTS="-Dcassandra.jmx.local.port=${jmx_port}" and not JMX_OPTS="-Djava.rmi.server.hostname=localhost"


Basically when you use JMX_OPTS from .new (JMX_OPTS="-Dcassandra.jmx.local.port=${jmx_port}) storage node is down in UI, but this issue is caused by not yet updated plugins on agent side. When the plugin is updated, storage node is up.

Other relevant issues from this bz are probably just following:
 - original cassandra-jvm.properties has jmx_port and heap and stack sizes without quotes
 - cassandra-jvm.properties.new has jmx_port and heap and stack sizes with quotes 
Quotes are causing bz1272239

Note You need to log in before you can comment on or make changes to this bug.