Created attachment 1505961 [details] metrics 36 logs Description of problem: Deployed metrics v3.6.173.0.138, hawkular-metrics pod is not READY # oc -n openshift-infra get pod NAME READY STATUS RESTARTS AGE hawkular-cassandra-1-ctc07 1/1 Running 0 38m hawkular-metrics-t9tz5 0/1 Running 2 18m heapster-zbz8g 0/1 Running 4 37m # oc -n openshift-infra logs hawkular-metrics-t9tz5 2018-11-15 06:40:32,319 INFO [org.wildfly.security] (MSC service thread 1-2) ELY00001: WildFly Elytron version 1.1.10.Final-redhat-1 2018-11-15 06:40:32,427 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "hawkular-metrics.war")]) - failure description: { "WFLYCTL0412: Required services that are not installed:" => [ "jboss.naming.context.java.jboss.infinispan.container.hawkular-metrics", "jboss.naming.context.java.jboss.infinispan.cache.hawkular-metrics.locks" ], "WFLYCTL0180: Services with missing/unavailable dependencies" => [ "jboss.naming.context.java.module.hawkular-metrics.hawkular-metrics.env.container.hawkular-metrics is missing [jboss.naming.context.java.jboss.infinispan.container.hawkular-metrics]", "jboss.naming.context.java.module.hawkular-metrics.hawkular-metrics.env.cache.locks is missing [jboss.naming.context.java.jboss.infinispan.cache.hawkular-metrics.locks]" ] } 2018-11-15 06:40:32,437 INFO [org.jboss.as.server] (ServerService Thread Pool -- 34) WFLYSRV0010: Deployed "hawkular-metrics.war" (runtime-name : "hawkular-metrics.war") 2018-11-15 06:40:32,437 INFO [org.jboss.as.server] (ServerService Thread Pool -- 34) WFLYSRV0010: Deployed "activemq-rar.rar" (runtime-name : "activemq-rar.rar") 2018-11-15 06:40:32,440 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report WFLYCTL0184: New missing/unsatisfied dependencies: service jboss.naming.context.java.jboss.infinispan.cache.hawkular-metrics.locks (missing) dependents: [service jboss.naming.context.java.module.hawkular-metrics.hawkular-metrics.env.cache.locks] service jboss.naming.context.java.jboss.infinispan.container.hawkular-metrics (missing) dependents: [service jboss.naming.context.java.module.hawkular-metrics.hawkular-metrics.env.container.hawkular-metrics] 2018-11-15 06:40:32,497 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server 2018-11-15 06:40:32,499 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management 2018-11-15 06:40:32,499 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990 2018-11-15 06:40:32,499 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: JBoss EAP 7.1.4.GA (WildFly Core 3.0.17.Final-redhat-1) started (with errors) in 8689ms - Started 424 of 709 services (22 services failed or missing dependencies, 391 services are lazy, passive or on-demand) Version-Release number of selected component (if applicable): # openshift version openshift v3.6.173.0.138 kubernetes v1.6.1+5115d708d7 etcd 3.2.1 metrics images version: v3.6.173.0.138 How reproducible: Always Steps to Reproduce: 1. Deploy metrics v3.6.173.0.138 2. 3. Actual results: hawkular-metrics pod is not READY Expected results: hawkular-metrics should be READY Additional info:
Created attachment 1505962 [details] v3.6.173.0.138_standalone.xml
Created attachment 1505963 [details] v3.6.173.0.130-3_standalone.xml
Created attachment 1506308 [details] patch file
The latest metrics-hawkular-metrics is metrics-hawkular-metrics:v3.6.173.0.139-1, issue is still not fixed. still see error in metrics-hawkular-metrics pod logs: 2018-11-19 13:10:24,926 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "hawkular-metrics.war")]) - failure description: { "WFLYCTL0412: Required services that are not installed:" => [ "jboss.naming.context.java.jboss.infinispan.container.hawkular-metrics", "jboss.naming.context.java.jboss.infinispan.cache.hawkular-metrics.locks" ], "WFLYCTL0180: Services with missing/unavailable dependencies" => [ "jboss.naming.context.java.module.hawkular-metrics.hawkular-metrics.env.container.hawkular-metrics is missing [jboss.naming.context.java.jboss.infinispan.container.hawkular-metrics]", "jboss.naming.context.java.module.hawkular-metrics.hawkular-metrics.env.cache.locks is missing [jboss.naming.context.java.jboss.infinispan.cache.hawkular-metrics.locks]" ] BTW, since we want to release openshift v3.6.173.0.138, so the metrics-hawkular-metrics should use v3.6.173.0.138 two, and the latest metrics-hawkular-metrics:v3.6.173.0.138 points to v3.6.173.0.138-2, then the image version fix this error should be v3.6.173.0.138-3
Created attachment 1507223 [details] metrics pods log - v3.6.173.0.139
Tested with metrics-hawkular-metrics-v3.6.173.0.139-2, issue is not fixed and hawkular-metrics threw out error: "org.jboss.weld.exceptions.DeploymentException: WELD-001408: Unsatisfied dependencies for type MetricsService with qualifiers @Default" more info please see the attached logs file # oc -n openshift-infra get pod NAME READY STATUS RESTARTS AGE hawkular-cassandra-1-kmrtw 1/1 Running 0 22m hawkular-metrics-rnjpw 0/1 Running 2 22m heapster-kvpvh 0/1 Running 2 22m
Created attachment 1507572 [details] metrics logs with metrics-hawkular-metrics-v3.6.173.0.139-2
Tested metrics-hawkular-metrics:v3.6.173.0.139-3, issue is fixed The most important thing is we want to release errata: https://errata.devel.redhat.com/advisory/37911 and openshift version is v3.6.173.0.138, so the metrics-hawkular-metrics should use v3.6.173.0.138 too, the latest metrics-hawkular-metrics:v3.6.173.0.138 links to v3.6.173.0.138-2, then the image version fix this error should be v3.6.173.0.138-3, we should re-tag metrics-hawkular-metrics:v3.6.173.0.139-3 as metrics-hawkular-metrics:v3.6.173.0.138-3 and I had tested, the following metrics v3.6.173.0.138 works with metrics-hawkular-metrics-v3.6.173.0.139-3 metrics-cassandra-v3.6.173.0.138-2 metrics-heapster-v3.6.173.0.138-2
Issue is fixed with images: metrics-hawkular-metrics:v3.6.173.0.139-3 metrics-cassandra:v3.6.173.0.139-1 metrics-heapster:v3.6.173.0.139-1 Set to VERIFIED
Can someone please add doc_text to this bug so I can add it to an RHSA that will be going out soon? Thanks!
I would say this doesn't need any doc update because this is not a bug that could be seen in any publicly released images, only in internal ones. I'll set the doc type to 'no doc update', let me know if that's wrong.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:3598