Bug 1466741

Summary: [paid][online-stg]no permissions for indices:data/read/field_stats on kibana UI
Product: OpenShift Online Reporter: Junqi Zhao <juzhao>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED CURRENTRELEASE QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.xCC: abhgupta, aos-bugs, ihorvath, juzhao
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-09 18:58:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Kibana -- no permissions for indices
none
Issue is fixed none

Description Junqi Zhao 2017-06-30 11:17:41 UTC 
Created attachment 1293190 [details]
Kibana -- no permissions for indices

Description of problem:
online-stg env, when logging in Kibana, no permissions for indices error showed, and after a few minutes, logs still can not be shown.

Discover: [security_exception] no permissions for indices:data/read/field_stats

Error: [security_exception] no permissions for indices:data/read/field_stats
ErrorAbstract@https://logs.online-stg.openshift.com/bundles/kibana.bundle.js?v=10000:64857:20
StatusCodeError@https://logs.online-stg.openshift.com/bundles/kibana.bundle.js?v=10000:65019:6
respond@https://logs.online-stg.openshift.com/bundles/kibana.bundle.js?v=10000:66224:16
checkRespForFailure@https://logs.online-stg.openshift.com/bundles/kibana.bundle.js?v=10000:66187:8
[24]</AngularConnector.prototype.request/<@https://logs.online-stg.openshift.com/bundles/kibana.bundle.js?v=10000:64805:8
processQueue@https://logs.online-stg.openshift.com/bundles/commons.bundle.js?v=10000:41883:29
scheduleProcessQueue/<@https://logs.online-stg.openshift.com/bundles/commons.bundle.js?v=10000:41899:28
$RootScopeProvider/this.$get</Scope.prototype.$eval@https://logs.online-stg.openshift.com/bundles/commons.bundle.js?v=10000:43127:17
$RootScopeProvider/this.$get</Scope.prototype.$digest@https://logs.online-stg.openshift.com/bundles/commons.bundle.js?v=10000:42938:16
$RootScopeProvider/this.$get</Scope.prototype.$apply@https://logs.online-stg.openshift.com/bundles/commons.bundle.js?v=10000:43235:14
done@https://logs.online-stg.openshift.com/bundles/commons.bundle.js?v=10000:37684:37
completeRequest@https://logs.online-stg.openshift.com/bundles/commons.bundle.js?v=10000:37882:8
requestLoaded@https://logs.online-stg.openshift.com/bundles/commons.bundle.js?v=10000:37823:10

Version-Release number of selected component (if applicable):

OpenShift Master:  v3.5.5.30 (online version 3.5.1.55)
Kubernetes Master: v1.5.2+43a9be4 

How reproducible:
Always

Steps to Reproduce:
1. Create one project and populate logs
2. login https://logs.online-stg.openshift.com/ to check los
3.

Actual results:
no permissions for indices error showed, and after a few minutes, logs still can not be shown.


Expected results:
logs could be retrieved from Kibana

Additional info:
Comment 1 Jeff Cantrill 2017-06-30 15:51:12 UTC 
This is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1456413
Comment 2 Junqi Zhao 2017-07-04 00:48:51 UTC 
I am not sure whether it is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1456413 or not.

But BZ #1456413 was fixed on Logging 3.6.0, the scenario is: Create a new project and don't deploy anything on it so that it's empty and then login Kibana UI.


BZ #1466741 was using logging 3.5 images,  since OpenShift Master version is  v3.5.5.30, and testing scenario is: Create one project and populate logs and then login Kibana UI. If we want to set this defect to ON_QA, I think we should backport the fix to logging 3.5 images.
Comment 3 Junqi Zhao 2017-07-06 07:15:31 UTC 
Tested today, this issue is still exists. 


OpenShift Master:
    v3.5.5.30 (online version 3.5.1.55)
Kubernetes Master:
    v1.5.2+43a9be4
Comment 5 Junqi Zhao 2017-07-07 02:50:48 UTC 
@Abhishek,
Checked, issue still exists since the current logging-elasticsearch image is not the same in Comment 4.

There are two ways if we want to verify it is fixed.

1. Scale all logging-elasticsearch dc to 0, and edit logging-elasticsearch dc to use images version 3.5.0-27, then scale up all logging-elasticsearch dc.

2. Redeploy logging, it will automatically pick up the current latest image logging-elasticsearch:3.5.0-27.

Please help to make these changes if we are allowed to do this, anything you are concerned about, please contact jcantril

Thanks
Comment 6 Junqi Zhao 2017-07-07 06:00:31 UTC 
@Abhishek,

Please ignore my Comment 5, we checked logging images on online-stg, and found logging images version is 3.4.1, it is not 3.5, generally speaking, OCP and logging version should be the same. I think we should deploy logging 3.5 on this env next time.

We also checked on pro-us-east1 env, OCP and logging version are 3.5, they are the same, and it does not have such defect on pro-us-east1 env. 


@jeff
Based on the previous comments, we don't need logging-elasticsearch:3.5.0-27 fix, sorry for the inconvenience.
Comment 7 ihorvath 2017-07-07 17:15:42 UTC 
I fixed the online-stg cluster so logging is also 3.5 on there.
Comment 8 Junqi Zhao 2017-07-10 00:50:32 UTC 
Defect is fixed and log entries could be found in Kibana UI, see the attached UI
Comment 9 Junqi Zhao 2017-07-10 00:51:03 UTC 
Created attachment 1295662 [details]
Issue is fixed