Bug 1519694

Summary: [3.5.1]Enabled ops cluser,log in kibana UI, there is not document log under .all index
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: high    
Version: 3.5.1CC: anli, aos-bugs, jcantril, jmalde, nhosoi, nnosenzo, rkant, rmeggins, smunilla, vlaad
Target Milestone: ---Keywords: Regression, Reopened
Target Release: 3.5.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of:
: 1519705 (view as bug list) Environment:
Last Closed: 2018-10-16 01:08:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1519705, 1526287    
Attachments:
Description Flags
enabled ops cluster, kibana UI, there is not result under .all index
none
enabled ops cluster, kibana UI, logs only could be found under individual project
none
enabled ops cluster, kibana UI, there are logs under .all index
none
enabled ops cluster, kibana-ops UI, there are logs under .all index
none
logs under .all index none

Description Junqi Zhao 2017-12-01 08:15:37 UTC
Created attachment 1361466 [details]
enabled ops cluster, kibana UI, there is not result under .all index

Description of problem:
Enabled ops cluser,log in kibana UI, there is not log entry under .all index, logs only could be found under separate project index.
Time range already selected "Last 7 days".

This issue only exist on ops cluster.

Version-Release number of selected component (if applicable):
Images:
logging-elasticsearch-3.5.0-58
logging-curator-v3.5.5.31.47-10
logging-fluentd-3.5.0-46
logging-kibana-3.5.0-51
logging-auth-proxy-3.5.0-45

# openshift version
openshift v3.5.5.31.47
kubernetes v1.5.2+43a9be4
etcd 3.1.0


How reproducible:
Always

Steps to Reproduce:
1. Deploy logging with enabled ops cluster.
2. Login kibana UI to check logs.
3.

Actual results:
There is not log entry under .all index

Expected results:
There should be log entries under .all index

Additional info:
# Inventory file
[OSEv3:children]
masters

[masters]
${MASTER} openshift_public_hostname=${MASTER}


[OSEv3:vars]
ansible_ssh_user=root
ansible_ssh_private_key_file="~/libra.pem"
deployment_type=openshift-enterprise

# Logging
openshift_logging_install_logging=true
openshift_logging_kibana_hostname=kibana.apps.${SUB_DOMAIN}
openshift_logging_kibana_ops_hostname=kibana-ops.apps.${SUB_DOMAIN}
public_master_url=https://${MASTER}:8443
openshift_logging_image_prefix=${IMAGE_PREFIX}
openshift_logging_image_version=v3.5
openshift_logging_namespace=logging
openshift_logging_elasticsearch_memory_limit=1Gi
openshift_logging_es_ops_memory_limit=512Mi
openshift_logging_use_ops=true

Comment 1 Junqi Zhao 2017-12-01 08:18:03 UTC
Created attachment 1361467 [details]
enabled ops cluster, kibana UI, logs only could be found under individual project

Comment 5 Junqi Zhao 2018-01-09 00:59:40 UTC
Issue is not fixed, it was changed to ON_QA by errata.
Enabled ops cluser, there is still no document log under .all index

Images:
logging-kibana/images/3.5.0-54
logging-curator/images/v3.5.5.31.58-1
logging-fluentd/images/3.5.0-51
logging-auth-proxy/images/3.5.0-47
logging-elasticsearch/images/3.5.0-61

Comment 10 Junqi Zhao 2018-03-22 08:26:54 UTC
Issue is fixed, please change to ON_QA

Enabled ops cluster, there are logs under .all index on kibana UI, see the attached pictures.


# openshift version
openshift v3.5.5.31.64
kubernetes v1.5.2+43a9be4
etcd 3.1.0

Images:
logging-curator-v3.5.5.31.64-1
logging-elasticsearch-3.5.0-62
logging-fluentd-3.5.0-53
logging-kibana-3.5.0-54
logging-auth-proxy-3.5.0-47

Comment 11 Junqi Zhao 2018-03-22 08:27:59 UTC
Created attachment 1411632 [details]
enabled ops cluster, kibana UI, there are logs under .all index

Comment 12 Junqi Zhao 2018-03-22 08:28:27 UTC
Created attachment 1411633 [details]
enabled ops cluster, kibana-ops UI, there are logs under .all index

Comment 14 Junqi Zhao 2018-10-08 10:19:11 UTC
Issue is fixed with logging images v3.5.5.31.74-1, there are document logs under .all index, see the attached picture

Comment 15 Junqi Zhao 2018-10-08 10:20:36 UTC
Created attachment 1491580 [details]
logs under .all index

Comment 21 Jeff Cantrill 2018-10-16 01:08:33 UTC
This is not a fix that is available in the fluent image but the elasticsearch image and requires v3.5.5.31.74-1 or better.  I see this image in the catalogue:

https://access.redhat.com/containers/#/registry.access.redhat.com/openshift3/logging-elasticsearch/images/v3.5.5.31.67-1

Closing per #c16