Created attachment 1471412 [details] user project index is not created Description of problem: free-stg cluster, ES nodes are all in YELLOW status, user project index is not created and could not be found at Kibana NAME READY STATUS RESTARTS AGE logging-es-data-master-dm4qr1as-29-kvppx 2/2 Running 0 2d logging-es-data-master-qhgbgiw5-24-qj48k 2/2 Running 0 2d logging-es-data-master-szybvfio-27-npjhj 2/2 Running 0 2d # oc logs -c elasticsearch logging-es-data-master-dm4qr1as-29-kvppx -n logging Clustername: logging-es Clusterstate: YELLOW Number of nodes: 3 Number of data nodes: 3 .searchguard.logging-es-data-master-dm4qr1as index already exists, so we do not need to create one. Populate config from /opt/app-root/src/sgconfig/ Will update 'config' with /opt/app-root/src/sgconfig/sg_config.yml SUCC: Configuration for 'config' created or updated Will update 'roles' with /opt/app-root/src/sgconfig/sg_roles.yml SUCC: Configuration for 'roles' created or updated Will update 'rolesmapping' with /opt/app-root/src/sgconfig/sg_roles_mapping.yml SUCC: Configuration for 'rolesmapping' created or updated Will update 'internalusers' with /opt/app-root/src/sgconfig/sg_internal_users.yml SUCC: Configuration for 'internalusers' created or updated Will update 'actiongroups' with /opt/app-root/src/sgconfig/sg_action_groups.yml SUCC: Configuration for 'actiongroups' created or updated Done with success Version-Release number of selected component (if applicable): logging-elasticsearch:v3.10.14 How reproducible: Always Steps to Reproduce: 1. Check ES logs 2. 3. Actual results: Expected results: Additional info:
waited for 2 hours, user project index could not be created Error in Kibana Discover: [exception] The index 'project.empty-project.*' was not found. This could mean data has not yet been collected. Error: [exception] The index 'project.empty-project.*' was not found. This could mean data has not yet been collected. ErrorAbstract@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:95501:20 StatusCodeError@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:95663:6 respond@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:96868:16 checkRespForFailure@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:96831:8 [24]</AngularConnector.prototype.request/<@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:95449:8 processQueue@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:42452:29 scheduleProcessQueue/<@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:42468:28 $eval@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:43696:17 $digest@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:43507:16 $apply@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:43804:14 done@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:38253:37 completeRequest@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:38451:8 requestLoaded@https://logs.free-stg.openshift.com/bundles/commons.bundle.js?v=10229:38392:10
Created attachment 1472873 [details] Error in kibana UI
@Justin This cluster is green but only has fluent on 4 nodes. The screen shot displays the pod and namespace which is running on a node that does not have fluentd. # oc get pod -n tcs15 -o wide NAME READY STATUS RESTARTS AGE IP NODE ruby-ex-1-2724w 1/1 Running 0 11h 10.130.12.210 ip-172-31-65-66.us-east-2.compute.internal ruby-ex-1-build 0/1 Completed 0 11h 10.130.12.209 ip-172-31-65-66.us-east-2.compute.internal [root@free-stg-master-03fb6 logging]# oc get node -l logging-infra-fluentd=true NAME STATUS ROLES AGE VERSION ip-172-31-65-74.us-east-2.compute.internal Ready compute,master 1y v1.10.0+b81c8f8 ip-172-31-69-53.us-east-2.compute.internal Ready compute,infra 274d v1.10.0+b81c8f8 ip-172-31-75-47.us-east-2.compute.internal Ready compute,infra 274d v1.10.0+b81c8f8 ip-172-31-78-254.us-east-2.compute.internal Ready master 1y v1.10.0+b81c8f8 I can manually label all or the compute nodes but we I'm deferring doing that given we probably want to modify the inventory. How do we make that happen.
Blocked by Bug 1618577
Could show user project index in Kibana UI now logging images version: v3.10.27 OpenShift Master:v3.10.27 Kubernetes Master:v1.10.0+b81c8f8 OpenShift Web Console:v3.10.27