Bug 1467834 - .all index is missing when ops cluster is enabled
.all index is missing when ops cluster is enabled
Status: CLOSED WORKSFORME
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging (Show other bugs)
3.6.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Jeff Cantrill
Xia Zhao
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-05 05:24 EDT by Xia Zhao
Modified: 2017-08-16 15 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-11 02:16:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
inventory file used for logging deployment (748 bytes, text/plain)
2017-07-05 05:24 EDT, Xia Zhao
no flags Details
es log (26.12 KB, text/plain)
2017-07-05 05:24 EDT, Xia Zhao
no flags Details
es ops log (21.76 KB, text/plain)
2017-07-05 05:25 EDT, Xia Zhao
no flags Details
ops logging UI (258.11 KB, image/png)
2017-07-05 05:26 EDT, Xia Zhao
no flags Details
non-ops logging UI (245.18 KB, image/png)
2017-07-05 05:26 EDT, Xia Zhao
no flags Details
ops disabled cluster, .all index on Kibana UI (228.31 KB, image/png)
2017-07-10 03:10 EDT, Junqi Zhao
no flags Details

  None (edit)
Description Xia Zhao 2017-07-05 05:24:04 EDT
Created attachment 1294520 [details]
inventory file used for logging deployment

Description of problem:
.all index is missing when ops cluster is enabled.
Issue did not exist if ops cluster is disabled.

Version-Release number of selected component (if applicable):
# openshift version
openshift v3.6.133
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

# rpm -qa | grep ansible
openshift-ansible-callback-plugins-3.6.133-1.git.0.950bb48.el7.noarch
openshift-ansible-docs-3.6.133-1.git.0.950bb48.el7.noarch
openshift-ansible-lookup-plugins-3.6.133-1.git.0.950bb48.el7.noarch
openshift-ansible-filter-plugins-3.6.133-1.git.0.950bb48.el7.noarch
openshift-ansible-playbooks-3.6.133-1.git.0.950bb48.el7.noarch
ansible-2.2.3.0-1.el7.noarch
openshift-ansible-3.6.133-1.git.0.950bb48.el7.noarch
openshift-ansible-roles-3.6.133-1.git.0.950bb48.el7.noarch

# Images tested with
${brew_registry}/openshift3/logging-curator         v3.6                3bc7ecc4f17d        About an hour ago   222.1 MB
${brew_registry}/openshift3/logging-kibana          v3.6                8da2ca80b6a4        2 hours ago         342.4 MB
${brew_registry}/openshift3/logging-elasticsearch   v3.6                dc20042569d0        2 hours ago         404.2 MB
${brew_registry}/openshift3/logging-auth-proxy      v3.6                bddefb8b639a        2 hours ago         214.8 MB
${brew_registry}/openshift3/logging-fluentd         v3.6                eed0e8e709c0        2 hours ago         231.5 MB


How reproducible:
Always

Steps to Reproduce:
1.Deploy logging v3.6 with ops cluster enabled
2.Wait until EFK are all on, and see "update_mapping" in both es and es-ops logs
$ oc get po
NAME                                          READY     STATUS    RESTARTS   AGE
logging-curator-1-8r3gr                       1/1       Running   0          1h
logging-curator-ops-1-kr440                   1/1       Running   0          1h
logging-es-data-master-3137qzx4-1-j464p       1/1       Running   0          1h
logging-es-ops-data-master-n3mjpg36-1-bs58q   1/1       Running   0          1h
logging-fluentd-588rp                         1/1       Running   0          1h
logging-fluentd-vjcm8                         1/1       Running   0          1h
logging-kibana-1-zzqt1                        2/2       Running   0          1h
logging-kibana-ops-1-qh5xs                    2/2       Running   0          1h
3.Login the non-ops logging UI --> see user indices there with log entries
4.Login the ops logging UI --> only .operations* exist

Actual results:
.all index is missing

Expected results:
.all index should exist

Additional info:
screenshots attached
es and es-ops logs attached
inventory file attached
Comment 1 Xia Zhao 2017-07-05 05:24 EDT
Created attachment 1294522 [details]
es log
Comment 2 Xia Zhao 2017-07-05 05:25 EDT
Created attachment 1294523 [details]
es ops log
Comment 3 Xia Zhao 2017-07-05 05:26 EDT
Created attachment 1294524 [details]
ops logging UI
Comment 4 Xia Zhao 2017-07-05 05:26 EDT
Created attachment 1294525 [details]
non-ops logging UI
Comment 5 Xia Zhao 2017-07-05 05:31:34 EDT
Closing since finally I get the .all index on the non-ops kibana UI after seeing "update_mapping" in es log for another 1 hour.
Comment 6 Junqi Zhao 2017-07-10 02:10:19 EDT
Tested today, .all index is not found on Kibana UI

# rpm -qa | grep openshift-ansible
openshift-ansible-3.6.139-1.git.0.4ff49c6.el7.noarch
openshift-ansible-roles-3.6.139-1.git.0.4ff49c6.el7.noarch
openshift-ansible-docs-3.6.139-1.git.0.4ff49c6.el7.noarch
openshift-ansible-lookup-plugins-3.6.139-1.git.0.4ff49c6.el7.noarch
openshift-ansible-callback-plugins-3.6.139-1.git.0.4ff49c6.el7.noarch
openshift-ansible-playbooks-3.6.139-1.git.0.4ff49c6.el7.noarch
openshift-ansible-filter-plugins-3.6.139-1.git.0.4ff49c6.el7.noarch

Images from brew registry
logging-elasticsearch   v3.6.138-1          2b2b061286e6        47 hours ago        404.2 MB
logging-kibana          v3.6.138-1          43aea64f70a8        47 hours ago        342.4 MB
logging-curator         v3.6.138-1          bf5d8756dcfc        47 hours ago        221.5 MB
logging-fluentd         v3.6.138-1          e4ed3ac61d69        47 hours ago        231.5 MB
logging-auth-proxy      v3.6.138-1          9b98fea74082        47 hours ago        214.8 MB
Comment 7 Junqi Zhao 2017-07-10 03:08:53 EDT
ops disabled cluster, kibana UI have .all index, see the attached UI. 
bz #1467834 is only happened with ops enabled cluster.
Comment 8 Junqi Zhao 2017-07-10 03:10 EDT
Created attachment 1295707 [details]
ops disabled cluster, .all index on Kibana UI
Comment 9 Xia Zhao 2017-07-11 02:16:19 EDT
It works for me again after waiting for long,  similar with those in comment #5.
I'm not able to see .all index on OCP 3.6 with these v3.6 images deployed:

openshift3/logging-auth-proxy    4cf6b1d60d2b
openshift3/logging-kibana    4563b27eac07
openshift3/logging-elasticsearch    8809f390a819
openshift3/logging-fluentd    a2ea005ef4f6
openshift3/logging-curator    ea1887b8e441
Comment 10 Xia Zhao 2017-07-11 02:16:52 EDT
s/not/now/ in comment #9

Note You need to log in before you can comment on or make changes to this bug.