Bug 1933978 - Unable to access/use index pattern after a couple of days [NEEDINFO]
Summary: Unable to access/use index pattern after a couple of days
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.5
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 4.6.z
Assignee: ewolinet
QA Contact: Anping Li
URL:
Whiteboard: logging-exploration
: 1902112 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-02 07:59 UTC by Simon Reber
Modified: 2021-10-25 05:17 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-12 12:16:04 UTC
Target Upstream Version:
periklis: needinfo? (kiyyappa)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 2106 0 None open Bug 1933978: Update OD security dependency to resolve kibana index migration issue 2021-04-16 17:25:53 UTC
Red Hat Knowledge Base (Solution) 5885501 0 None None None 2021-03-16 08:41:59 UTC
Red Hat Product Errata RHBA-2021:1489 0 None None None 2021-05-12 12:16:17 UTC

Description Simon Reber 2021-03-02 07:59:55 UTC
Description of problem:

After addressing https://bugzilla.redhat.com/show_bug.cgi?id=1895581 it was possible to create index pattern for all users. This is also working ever since but now there is a new problem.

After a couple of days when the index pattern was created, the user is again redirected (upon login) by Kibana to the page, to create an index pattern (even though it still should exist).

Interesting enough, in the Saved Objects overview the `app` index pattern is reported and visible. But it can't be viewed, modified or even removed. So the user is again pretty much blocked as Kibana does not allow to run activity as it does expect an index pattern being created but also has problem with the one that already exists.

Also users are reporting error messages such as "Saved object is missing" and "Could not locate that index-pattern".

Restarting Kibana does not help as the problem persists. The only work-around currently is to remove the user indices as afterwards, the user is again able to create an index pattern (this is working) and can also use Kibana normally (but again after several days the issue comes again).

Also generally the user indices look OK and I suspect that it's rather a problem with permissions that the user is unable to access the user indices or similar

Version-Release number of selected component (if applicable):

 - clusterlogging.4.5.0-202102032236.p0
 - elasticsearch-operator.4.5.0-202102041049.p0

How reproducible:

 - N/A

Steps to Reproduce:
1. Not yet been able to reproduce

Actual results:

Users are redirected to index pattern creation page and is unable to use Kibana as the index pattern does exist respectively can't be accessed for some reason

Expected results:

Kibana to work continuously without any issue

Additional info:

Comment 33 Anping Li 2021-03-31 12:02:11 UTC
By accident, I reproduced this issue with following steps. 
1) Deploy Cluster Logging 4.4.
2) Upgrade Logging to 4.5.36  --> All works well
3) Delete CLO operators and EO operators
4) Install operator 4.5.33.   --> Save

Comment 35 Anping Li 2021-03-31 14:00:38 UTC
# Kibana popup message below when click the index pattern.
Could not locate that index-pattern (id: 468eab30-920c-11eb-ba63-91f02039c02d), click here to re-create it 

#Delete the saved objects(Index pattern), report 403.

{"type":"response","@timestamp":"2021-03-31T13:48:26Z","tags":[],"pid":121,"method":"delete","statusCode":403,"req":{"url":"/api/saved_objects/index-pattern/468eab30-920c-11eb-ba63-91f02039c02d","method":"delete","headers":{"host":"kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com","user-agent":"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0","accept":"*/*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.5","content-type":"application/json","forwarded":"for=61.51.136.151;host=kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com;proto=https","kbn-version":"6.8.1","origin":"https://kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com","referer":"https://kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com/app/kibana","x-forwarded-access-token":"sha256~FXL-wh_YXqNJe2_1DaE7zkMxd7l0NE-xgdgkb2u6fXA","x-forwarded-email":"testuser-0@cluster.local","x-forwarded-for":"61.51.136.151, 10.131.0.5","x-forwarded-host":"kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-user":"testuser-0","securitytenant":"__user__"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com/app/kibana"},"res":{"statusCode":403,"responseTime":36,"contentLength":9},"message":"DELETE /api/saved_objects/index-pattern/468eab30-920c-11eb-ba63-91f02039c02d 403 36ms - 9.0B"}

#Create index pattern, report 403.

{"type":"response","@timestamp":"2021-03-31T13:56:55Z","tags":[],"pid":121,"method":"post","statusCode":403,"req":{"url":"/api/saved_objects/index-pattern?overwrite=false","method":"post","headers":{"host":"kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com","user-agent":"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0","content-length":"66","accept":"*/*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.5","content-type":"application/json","forwarded":"for=61.51.136.151;host=kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com;proto=https","kbn-version":"6.8.1","origin":"https://kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com","referer":"https://kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com/app/kibana","x-forwarded-access-token":"sha256~FXL-wh_YXqNJe2_1DaE7zkMxd7l0NE-xgdgkb2u6fXA","x-forwarded-email":"testuser-0@cluster.local","x-forwarded-for":"61.51.136.151, 10.128.2.10","x-forwarded-host":"kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-user":"testuser-0","securitytenant":"__user__"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://kibana-openshift-logging.apps.anli461.qe.gcp.devcluster.openshift.com/app/kibana"},"res":{"statusCode":403,"responseTime":27,"contentLength":9},"message":"POST /api/saved_objects/index-pattern?overwrite=false 403 27ms - 9.0B"}



Migrate from 4.5.36 to 4.5.34 is not a real custom case. I will try to get a 100% reproduce way.

Comment 44 Anping Li 2021-04-29 23:27:30 UTC
Verified on CSV elasticsearch-operator.4.6.0-202104282223.p0.

Comment 46 Matthew Robson 2021-05-03 17:05:22 UTC
*** Bug 1902112 has been marked as a duplicate of this bug. ***

Comment 50 errata-xmlrpc 2021-05-12 12:16:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.28 extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1489

Comment 54 Periklis Tsirakidis 2021-10-21 13:39:16 UTC
*** Bug 2005589 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.