Description of problem: Customer is deploying new metrics components using [0] and cassandra is failing consistently. Logs from a currently trying to run pod as well as a previous pod (oc logs <POD> -p) will be attached shortly. [0] oc new-app --as=system:serviceaccount:openshift-infra:metrics-deployer -f metrics-deployer.yaml -p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.management-ocp-nonprod-int.uscis.dhs.gov -p USE_PERSISTENT_STORAGE=false --loglevel=8 Version-Release number of selected component (if applicable): 3.4.0
Please have them update the version of metrics to be installed to 3.4.1. The 3.4.1 image have the bug fixes required against the 3.4 version of OpenShift. This can be done by setting the IMAGE_VERSION property or modifying the default value in the deployer's template.
Can you also attach the output of 'oc get pods -o yaml -n openshift-infra' And the metrics-deployer.yaml file that they are using? If they are using 3.4, it should look something like https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_hosted_templates/files/v1.4/enterprise/metrics-deployer.yaml
Comparing the metrics-deployer.yaml file attached (privately) vs the github version provided in comment 5 I only see a few differences off the bat: first is the IMAGE_VERSION. The file attached specifically points to 3.4.0 but the github version points to v3.4 Second is the METRIC_RESOLUTION. The file attached specifies 15s but the github version has 30s set Looking onto the pods yaml, we can see that cassandra has restarted 98 times since deploy (I believe occurred on Friday (2017/04/28))
The output of 'oc get pods -o yaml -n openshift-infra' is correct. Which makes this even more confusing. The reason the pod is getting restarted is because the post start script is being killed with an exit status of 126: "Killing container with docker id XXXXXXXXXXXXXXXX: PostStart handler: Error executing in Docker Container: 126" A 126 error code usually indicates that the file exists but it cannot execute the binary. I have tried to reproduce this problem locally and it works fine for me. @lizhou: can you see if this is reproducible for you as well? Is there anything else about this cluster which may help us determine what is going on? Are they using the default security context, or have those been changed?
@Matt, issue was not reproduced with Metrics 3.4.0/3.4.1 in my environment # docker images | grep metrics openshift3/metrics-hawkular-metrics 3.4.1 7d4bd715f9ad 6 days ago 1.261 GB openshift3/metrics-heapster 3.4.1 0d79f78e5371 9 days ago 318 MB openshift3/metrics-cassandra 3.4.1 08fdb9958866 9 days ago 539.5 MB openshift3/metrics-deployer 3.4.1 8ed9217d0c54 9 days ago 864.2 MB openshift3/metrics-deployer 3.4.0 57baaa1c797f 10 weeks ago 862.9 MB openshift3/metrics-heapster 3.4.0 04c115b270b4 3 months ago 317.8 MB openshift3/metrics-cassandra 3.4.0 b5d700281ef7 4 months ago 649.2 MB openshift3/metrics-hawkular-metrics 3.4.0 ef113cd9dc4a 4 months ago 1.508 GB
The only thing I can think of would be an issue with something strange like some security context which prevents the scripts from being executed due to some permission issue. A work around for this would be to remove the poststart hook from the RC for Cassandra. That should allow the pod to start up properly. Since they are not using persistent data, they don't need the poststart hook to update any existing data.
@ejones: do you know if they are using the default security context with their installation? If they remove the poststart script, does Cassandra start up properly?
@Matt, It appears that we are no longer seeing this issue after upgrading the cluster to 3.4.1 per your earlier recommendation. Seems we are now having issues with certs and routes and such but I believe we do not have the ability to test this bz any further in the customer's cluster so if you guys are unable to reproduce, we should likely close this bug.
Ok, as we do not have the ability to reproduce this issue and an update to the latest bug fix version fixed the issue for the user, we are closing this issue as 'INSUFFICIENT_DATA'
I am marking this as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1471251 *** This bug has been marked as a duplicate of bug 1471251 ***