Bug 1576665 - hawkular-metrics pod failed to start up, threw out "java.io.IOException: Invalid secret key format"
Summary: hawkular-metrics pod failed to start up, threw out "java.io.IOException: Inva...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Hawkular
Version: 3.4.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 3.4.z
Assignee: Ruben Vargas Palma
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks: 1479992
TreeView+ depends on / blocked
 
Reported: 2018-05-10 06:08 UTC by Junqi Zhao
Modified: 2019-11-21 18:37 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-21 18:37:56 UTC
Target Upstream Version:


Attachments (Terms of Use)
metrics pods log (25.71 KB, application/x-gzip)
2018-05-10 06:10 UTC, Junqi Zhao
no flags Details

Description Junqi Zhao 2018-05-10 06:08:21 UTC
Description of problem:
hawkular-metrics pod failed to start up, threw out "java.io.IOException: Invalid secret key format"

Images
metrics-deployer/images/v3.4.1.44.54-1
metrics-hawkular-metrics/images/v3.4.1.44.54-1
metrics-heapster/images/v3.4.1.44.54-1
metrics-cassandra/images/v3.4.1.44.38-11

# oc get po
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-6wed5   1/1       Running   0          13m
hawkular-metrics-a9biz       0/1       Running   1          13m
heapster-cf8vm               0/1       Running   1          13m
metrics-deployer-zpbxd       1/1       Running   0          13m

05:20:12,895 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.jgroups.channel.ee: org.jboss.msc.service.StartException in service jboss.jgroups.channel.ee: java.security.PrivilegedActionException: java.lang.Exception: Unable to load keystore hawkular-jgroups.keystore: java.io.IOException: Invalid secret key format
	at org.wildfly.clustering.jgroups.spi.service.ChannelBuilder.start(ChannelBuilder.java:80)
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:2032)
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1955)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.PrivilegedActionException: java.lang.Exception: Unable to load keystore hawkular-jgroups.keystore: java.io.IOException: Invalid secret key format
	at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:865)
	at org.jboss.as.clustering.jgroups.JChannelFactory.createChannel(JChannelFactory.java:106)
	at org.wildfly.clustering.jgroups.spi.service.ChannelBuilder.start(ChannelBuilder.java:78)
	... 5 more
Caused by: java.lang.Exception: Unable to load keystore hawkular-jgroups.keystore: java.io.IOException: Invalid secret key format
	at org.jgroups.protocols.SYM_ENCRYPT.readSecretKeyFromKeystore(SYM_ENCRYPT.java:102)
	at org.jgroups.protocols.SYM_ENCRYPT.init(SYM_ENCRYPT.java:72)
	at org.jgroups.stack.ProtocolStack.initProtocolStack(ProtocolStack.java:861)
	at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:480)
	at org.jgroups.JChannel.init(JChannel.java:852)
	at org.jgroups.JChannel.<init>(JChannel.java:159)
	at org.jboss.as.clustering.jgroups.JChannelFactory$1.run(JChannelFactory.java:100)
	at org.jboss.as.clustering.jgroups.JChannelFactory$1.run(JChannelFactory.java:92)
	at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:863)
	... 7 more


used images from registry.access.redhat.com/openshift3/, it does not have this issue, it seems it is the metrics-deployer image issue
metrics-deployer/images/v3.4.1.44.53-3
metrics-hawkular-metrics/images/v3.4.1.44.53-2
metrics-heapster/images/v3.4.1.44.53-2
metrics-cassandra/images/v3.4.1.44.38-11


Version-Release number of selected component (if applicable):
See the description part

How reproducible:
Always

Steps to Reproduce:
1. Deploy metrics 3.4
2.
3.

Actual results:
hawkular-metrics pod failed to start up, threw out "java.io.IOException: Invalid secret key format"

Expected results:
All pods should be healthy

Additional info:

Comment 1 Junqi Zhao 2018-05-10 06:09:16 UTC
Blocks metrics 3.4 testing

Comment 2 Junqi Zhao 2018-05-10 06:10:31 UTC
Created attachment 1434201 [details]
metrics pods log

Comment 3 John Sanda 2018-05-30 14:34:46 UTC
Based on https://developer.jboss.org/thread/277727 I think we want to upgrade the base image to EAP 7.1.

Comment 5 Junqi Zhao 2018-06-19 04:11:14 UTC
Please change to ON_QA

Tested with metrics-hawkular-metrics/images/v3.4.1.44.56-1, issue is fixed
# oc get po -n openshift-infra
NAME                         READY     STATUS      RESTARTS   AGE
hawkular-cassandra-1-mx92r   1/1       Running     0          1h
hawkular-metrics-wsrmz       1/1       Running     0          1h
heapster-t2hwd               1/1       Running     0          1h
metrics-deployer-nt4p6       0/1       Completed   0          1h

other images are:
metrics-deployer/images/v3.4.1.44.55-1
metrics-heapster/images/v3.4.1.44.55-1
metrics-cassandra/images/v3.4.1.44.38-11

Comment 6 Junqi Zhao 2018-06-20 00:22:00 UTC
Set it to VERIFIED per Comment 5


Note You need to log in before you can comment on or make changes to this bug.