Bug 1511925
| Summary: | Can not login Kibana, Kibana error:[security_exception] no permissions for indices:data/read/mget | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | ||||||||
| Component: | Logging | Assignee: | Rich Megginson <rmeggins> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Junqi Zhao <juzhao> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | high | ||||||||||
| Version: | 3.5.1 | CC: | aos-bugs, nhosoi, rmeggins | ||||||||
| Target Milestone: | --- | Keywords: | Regression | ||||||||
| Target Release: | 3.5.z | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||||||
| Doc Text: |
I don't think a doc update is required because the broken code was never shipped to customers.
|
Story Points: | --- | ||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2017-12-14 21:02:32 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
Junqi Zhao
2017-11-10 12:45:34 UTC
Created attachment 1350461 [details]
logging dump output
Hi @Junqi,
Found these warnings in logging-20171110_072210/project:
29m 29m 1 logging-kibana-1-wzgr5 Pod Warning FailedMount {kubelet host-8-241-5.host.centralci.eng.rdu2.redhat.com} MountVolume.SetUp failed for volume "kubernetes.io/secret/c4d78e40-c60d-11e7-a33d-fa163ef17798-kibana" (spec.Name: "kibana") pod "c4d78e40-c60d-11e7-a33d-fa163ef17798" (UID: "c4d78e40-c60d-11e7-a33d-fa163ef17798") with: secrets "logging-kibana" not found
29m 29m 8 logging-kibana-1 ReplicationController Warning FailedCreate {replication-controller } Error creating: pods "logging-kibana-1-" is forbidden: service account logging/aggregated-logging-kibana was not found, retry after the service account is created
28m 28m 1 logging-kibana DeploymentConfig Warning FailedCreate {logging-kibana-1-deploy } Error creating: pods "logging-kibana-1-" is forbidden: service account logging/aggregated-logging-kibana was not found, retry after the service account is created
Could you check any openshift related errors in the system log?
If you search the string [1] with google, you'll find quite a number of people ran into the problem and some of them reported it was a resource issue, e.g., more memory was needed. I'm hoping the system log could contain any clue.
[1] - mountvolume setup failed for volume kubernetes io secret
> Could you check any openshift related errors in the system log?
>
> If you search the string [1] with google, you'll find quite a number of
> people ran into the problem and some of them reported it was a resource
> issue, e.g., more memory was needed. I'm hoping the system log could
> contain any clue.
>
> [1] - mountvolume setup failed for volume kubernetes io secret
I don't think it is related to resource, if it has something to do with resource, the kibana pod can not be in running status. And we did not have this issue before.
Error creating: pods "logging-kibana-1-" is forbidden: service account logging/aggregated-logging-kibana was not found, retry after the service account is created
I think it was try to find sa logging/aggregated-logging-kibana when this sa is not ready, and after a while, this sa is created, so it will not throw this warning then.
# oc get sa -n logging
NAME SECRETS AGE
aggregated-logging-curator 2 3h
aggregated-logging-elasticsearch 2 3h
aggregated-logging-fluentd 2 3h
aggregated-logging-kibana 2 3h
builder 2 3h
default 2 3h
deployer 2 3h
# journalctl | grep -i error | grep -i mount | grep -i "kibana"
found "invalid container name" info, such as
Nov 13 00:42:28 host-8-241-68.host.centralci.eng.rdu2.redhat.com atomic-openshift-node[26871]: I1113 00:42:28.546780 26871 factory.go:104] Error trying to work out if we can handle /system.slice/var-lib-origin-openshift.local.volumes-pods-d6c8c186\x2dc815\x2d11e7\x2da6f0\x2dfa163e365aa0-volumes-kubernetes.io\x7esecret-aggregated\x2dlogging\x2dkibana\x2dtoken\x2dpzx0r.mount: invalid container name
Nov 13 00:42:28 host-8-241-68.host.centralci.eng.rdu2.redhat.com atomic-openshift-node[26871]: I1113 00:42:28.547482 26871 factory.go:104] Error trying to work out if we can handle /system.slice/var-lib-origin-openshift.local.volumes-pods-d6c8c186\x2dc815\x2d11e7\x2da6f0\x2dfa163e365aa0-volumes-kubernetes.io\x7esecret-kibana.mount: invalid container name
Created attachment 1351388 [details]
journal log
https://github.com/openshift/origin-aggregated-logging/pull/784 koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=625144 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.5-rhel-7-docker-candidate-37544-20171114225315 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:3.5.0 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:3.5.0-56 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.5 Issue is fixed, there is not Kibana error:[security_exception] no permissions for indices:data/read/mget now. but a regression bug is found: https://bugzilla.redhat.com/show_bug.cgi?id=1513284 images: logging-elasticsearch/images/3.5.0-56 logging-kibana/images/3.5.0-51 logging-fluentd/images/3.5.0-46 logging-auth-proxy/images/3.5.0-45 logging-curator/images/v3.5.5.31.47-8 # openshift version openshift v3.5.5.31.47 kubernetes v1.5.2+43a9be4 etcd 3.1.0 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3438 |