Hide Forgot
Description of problem: hawkular-metrics pod failed to start up, threw out "java.io.IOException: Invalid secret key format" Images metrics-deployer/images/v3.4.1.44.54-1 metrics-hawkular-metrics/images/v3.4.1.44.54-1 metrics-heapster/images/v3.4.1.44.54-1 metrics-cassandra/images/v3.4.1.44.38-11 # oc get po NAME READY STATUS RESTARTS AGE hawkular-cassandra-1-6wed5 1/1 Running 0 13m hawkular-metrics-a9biz 0/1 Running 1 13m heapster-cf8vm 0/1 Running 1 13m metrics-deployer-zpbxd 1/1 Running 0 13m 05:20:12,895 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.jgroups.channel.ee: org.jboss.msc.service.StartException in service jboss.jgroups.channel.ee: java.security.PrivilegedActionException: java.lang.Exception: Unable to load keystore hawkular-jgroups.keystore: java.io.IOException: Invalid secret key format at org.wildfly.clustering.jgroups.spi.service.ChannelBuilder.start(ChannelBuilder.java:80) at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:2032) at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1955) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.security.PrivilegedActionException: java.lang.Exception: Unable to load keystore hawkular-jgroups.keystore: java.io.IOException: Invalid secret key format at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:865) at org.jboss.as.clustering.jgroups.JChannelFactory.createChannel(JChannelFactory.java:106) at org.wildfly.clustering.jgroups.spi.service.ChannelBuilder.start(ChannelBuilder.java:78) ... 5 more Caused by: java.lang.Exception: Unable to load keystore hawkular-jgroups.keystore: java.io.IOException: Invalid secret key format at org.jgroups.protocols.SYM_ENCRYPT.readSecretKeyFromKeystore(SYM_ENCRYPT.java:102) at org.jgroups.protocols.SYM_ENCRYPT.init(SYM_ENCRYPT.java:72) at org.jgroups.stack.ProtocolStack.initProtocolStack(ProtocolStack.java:861) at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:480) at org.jgroups.JChannel.init(JChannel.java:852) at org.jgroups.JChannel.<init>(JChannel.java:159) at org.jboss.as.clustering.jgroups.JChannelFactory$1.run(JChannelFactory.java:100) at org.jboss.as.clustering.jgroups.JChannelFactory$1.run(JChannelFactory.java:92) at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:863) ... 7 more used images from registry.access.redhat.com/openshift3/, it does not have this issue, it seems it is the metrics-deployer image issue metrics-deployer/images/v3.4.1.44.53-3 metrics-hawkular-metrics/images/v3.4.1.44.53-2 metrics-heapster/images/v3.4.1.44.53-2 metrics-cassandra/images/v3.4.1.44.38-11 Version-Release number of selected component (if applicable): See the description part How reproducible: Always Steps to Reproduce: 1. Deploy metrics 3.4 2. 3. Actual results: hawkular-metrics pod failed to start up, threw out "java.io.IOException: Invalid secret key format" Expected results: All pods should be healthy Additional info:
Blocks metrics 3.4 testing
Created attachment 1434201 [details] metrics pods log
Based on https://developer.jboss.org/thread/277727 I think we want to upgrade the base image to EAP 7.1.
Please change to ON_QA Tested with metrics-hawkular-metrics/images/v3.4.1.44.56-1, issue is fixed # oc get po -n openshift-infra NAME READY STATUS RESTARTS AGE hawkular-cassandra-1-mx92r 1/1 Running 0 1h hawkular-metrics-wsrmz 1/1 Running 0 1h heapster-t2hwd 1/1 Running 0 1h metrics-deployer-nt4p6 0/1 Completed 0 1h other images are: metrics-deployer/images/v3.4.1.44.55-1 metrics-heapster/images/v3.4.1.44.55-1 metrics-cassandra/images/v3.4.1.44.38-11
Set it to VERIFIED per Comment 5