Bug 1511432

This is a timing issue where the index-mappings are not corrected until the cache expires which occurs every minute.  After expiration, and possibly refreshing the browser, you should see the new index mappings.

Tested on free-stg cluster, created one project and waited for a long time, project index showed on kibana but disppeared soon, then it can not be found on kibana.

Image version is the same with this defect.

We don't have this issue a few days ago.

@jeff,  Hit Same error in a new environment with only 4 active projects. I can fetch data via curl command ( Both with cluster-admin and common user). I can list index with logging-curator

1) active projects (default,logging,anlitest,anlitest2)
2) curl -s -vk -H "X-Proxy-Remote-User: anli" -H "Authorization: Bearer GoMZeJrIo1ViauNJ6l0gz7HlMGHlNGatiYIc8lkKsLI" -H "X-Forwarded-For: 127.0.0.1" "https://logging-es.logging.svc:9200/project.anlitest*/_search?q=*" | python -c "import sys, json; print json.load(sys.stdin)['hits']['total']"
1444
3)  curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices
.kibana
.kibana.d033e22ae348aeb5660fc2140aec35850c4da997
.kibana.dfd6f6b721536c3de0c7718086ac4dd794c102d6
.searchguard.logging-es-data-master-8pbaz9xo
project.anlitest.13b84d89-c507-11e7-9d8a-fa163e78c39e.2017.11.10
project.anlitest2.300a3091-c5df-11e7-88cd-fa163e78c39e.2017.11.10
project.logging.8a782ab7-c471-11e7-bff3-fa163e78c39e.2017.11.10

I have waited for one hour and clear everything on Web browsers. the result is same with comment4.

3) indices list:

...
fluentd

This is bad.  This means the elasticsearch index name creation is having problems.  Sure enough, from the logs:

2017-11-09 03:13:47 -0500 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes.namespace_id field: {"docker"=>{"container_id"=>"f9cea645a380ea782d6efdc9ed423059b729c2903f2752519e87ff345f332053"}, "kubernetes"=>{"container_name"=>"caddy-docker-pod", "namespace_name"=>"ilf0m", "pod_name"=>"caddy-docker"}, ...

and

2017-11-09 04:11:39 -0500 [warn]: dump an error event: error_class=TypeError error="no implicit conversion of nil into String" tag="journal.container._openshift_" time=1510218699 record={"PRIORITY"=>"6", "_UID"=>"0", "_GID"=>"0", "_CAP_EFFECTIVE"=>"1fffffffff", "_SYSTEMD_SLICE"=>"system.slice", "_BOOT_ID"=>"a9da6ebbc15e4343a17cd9104a0fc503", "_MACHINE_ID"=>"321923336865459d9c8b86687ba3690d", "_HOSTNAME"=>"host-172-16-120-81", "_TRANSPORT"=>"journal", "_SYSTEMD_CGROUP"=>"/system.slice/docker.service", "_SYSTEMD_UNIT"=>"docker.service", "_SELINUX_CONTEXT"=>"system_u:system_r:container_runtime_t:s0", "_PID"=>"120481", "_COMM"=>"dockerd-current", "_EXE"=>"/usr/bin/dockerd-current", "_CMDLINE"=>"/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --selinux-enabled --log-driver=journald --signature-verification=False --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/rhel-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true --mtu=1350 --add-registry registry.reg-aws.openshift.com:443 --add-registry registry.access.redhat.com --block-registry registry.hacker.com --insecure-registry brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 --insecure-registry virt-openshift-05.lab.eng.nay.redhat.com:5000 --insecure-registry virt-openshift-05.lab.eng.nay.redhat.com:5001 --insecure-registry registry.reg-aws.openshift.com:443 --insecure-registry asb-registry.usersys.redhat.com:5000 --add-registry registry.access.redhat.com", "MESSAGE"=>"---> Installing application source ...", "CONTAINER_ID"=>"5db7208a666e", "CONTAINER_ID_FULL"=>"5db7208a666e7128943803e4c8c6a15d3e3449a0399bf5758ce98c0447d8b75f", "CONTAINER_NAME"=>"s2i_openshift_ruby_20_centos7_sha256_751a3cd1905914389fe568c25b3d5367cd705a0e4f81970a361f670ce891baf7_2563cbd7", "_SOURCE_REALTIME_TIMESTAMP"=>"1510218699134795", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.2.12.141", "ipaddr6"=>"fe80::858:aff:fe02:c8d", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2017-11-09T09:11:39.369525+00:00", "version"=>"0.12.39 1.6.0"}}}

There is same issue with latest v3.6 images. atomic-openshift-3.6.173.0.49 and Logging: v3.6.173.0.72

You can confirm that this is a regression?  This exact same test was run against 3.7.earlier and 3.6.earlier and there were no missing records/indices?  Because if the test creates and deletes namespaces, there should have been missing records/indices with previous releases.

When I log into Kibana as admin/admin, the first thing I see is that I can only select .all or .operations.*

I go to the Settings tab - Configure an index pattern - specify project.anlitest.* as the pattern.  It does some sort of search, because it thinks for a moment, then displays the Time-field name list with the correct field names in the list.  Select "@timestamp" and press Create.  This gives the following error:

Could not locate that index-pattern (id: project.anlitest.*)

But the index is clearly there:

oc exec -c elasticsearch $espod -- curl -s -k --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key https://localhost:9200/_cat/indices

green open project.anlitest.13b84d89-c507-11e7-9d8a-fa163e78c39e.2017.11.13 1 0  5328 0  1.4mb  1.4mb

I think what should happen is that there should be an index pattern called "project.*" which lists all of the projects.  That's what I see using the latest 3.7 deployment.  But it still won't let me create a new index pattern e.g. project.logging.*

Created attachment 1351670 [details]
video to show how to select, and remove, a namespace

I was able to manually create the "project.*" index in the QE lab kibana.

go to the Settings tab - Configure an index pattern - specify project.* as the pattern.  It does some sort of search, because it thinks for a moment, then displays the Time-field name list with the correct field names in the list.  Select "@timestamp" and press Create.

I think the severity of this bug should be decreased.  The workaround is pretty simple.  If you want to see the indices for a specific project, you can select the ".all" index, or use the "project.*" index pattern if it exists.  Then, scroll down to the "kubernetes.namespace_name" field and click on it. It should expand and show all of the namespaces that are included in the time window of the current search (by default, only the last 15 minutes).  Close to, and associated with, each namespace name, there is a magnifying class icon with a "-" and one with a "+".  If you click on the one with the "+" the view will change to show only records from that namespace.  The bar at the top will have that namespace name as the search criteria.  If you mouse over the namespace name in the bar, there will be several icons.  If you click on the trashcan icon, it will remove the namespace and show records from all namespaces again.

I have attached a short video demonstrating this.

I'll need to discuss with Jeff what the intention is of the new Kibana behavior.  This may be a new feature.  One of the problems with the old way of doing it is that it would overwhelm Kibana if you had hundreds of namespaces.  It is much easier to just show "project.*" by default and let the user specify which namespace(s) to search for and view, rather than via the index pattern dropdown selector.

I've removed the TestBlocker flag as I am not aware of any functionality which is prevented by this issue.

It might also not be a Regression, depending on if this new behavior is intentional.

@rich, @Jeff, I can create index pattern for projects following the comment 15. But notice that there is no default index for ordinary user in kibana (refer to the snapshot "No default index for ordinary user"). 

To fix this bug. I think the expected result in kibana is as following. Please correct me if it is wrong.

1) The .all index can be found for all users in kibana. the user can retrieve the documents using .all index. The documents are from all indices the user owned.

2) The .operation index can be found when log in as cluster-admin or cluster-reader in kibana. the user can retrieve logs which are generated by the default project, the openshift project, openshift-infra project and operation system log. 
   The .operation index is not in the kibana when login as non cluster users
   The .operation index is only in ops kibana when ops kibana was enabled.

3) All indices can be list in kibana if the user own this project and the project has generated logs.

@Anping.

Correction to item 1 of c#20.  The '.all' index is an alias that should be available only to users who are cluster-admin

@rmeggins
user can see the project indices by using workaround
https://bugzilla.redhat.com/show_bug.cgi?id=1511432#c15

But if there are a lot of projects, it will make customers disappointed if we let customers do the workaround manually

(In reply to Junqi Zhao from comment #23)
> @rmeggins
> user can see the project indices by using workaround
> https://bugzilla.redhat.com/show_bug.cgi?id=1511432#c15
> 
> But if there are a lot of projects, it will make customers disappointed if
> we let customers do the workaround manually

I mean if there are many projects, such as AA, BB, CC.., customers have to add project.AA*, project.BB*, .. indices to Kibana, if we jusut add project.* to Kibana, customers can not find every projects in Kibana

(In reply to Junqi Zhao from comment #24)
> (In reply to Junqi Zhao from comment #23)
> > @rmeggins
> > user can see the project indices by using workaround
> > https://bugzilla.redhat.com/show_bug.cgi?id=1511432#c15
> > 
> > But if there are a lot of projects, it will make customers disappointed if
> > we let customers do the workaround manually
> 
> I mean if there are many projects, such as AA, BB, CC.., customers have to
> add project.AA*, project.BB*, .. indices to Kibana, if we jusut add
> project.* to Kibana, customers can not find every projects in Kibana

I'm not sure what you mean.

If we add only project.* to Kibana, customers can find every project.  They can find all of the projects they have permissions to see.

The only thing we are taking away from customers is the ability to have, by default, a "project.namespacename.*" index pattern for all of the projects/namespaces they have permissions to view, in the drop-down selector in the upper left hand corner on the Discover tab page.

(In reply to Rich Megginson from comment #25)
> (In reply to Junqi Zhao from comment #24)
> > (In reply to Junqi Zhao from comment #23)
> > > @rmeggins
> > > user can see the project indices by using workaround
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1511432#c15
> > > 
> > > But if there are a lot of projects, it will make customers disappointed if
> > > we let customers do the workaround manually
> > 
> > I mean if there are many projects, such as AA, BB, CC.., customers have to
> > add project.AA*, project.BB*, .. indices to Kibana, if we jusut add
> > project.* to Kibana, customers can not find every projects in Kibana
> 
> I'm not sure what you mean.

See the attached picture " Snapshot 1", we have .all, project.logging.*, project.openshift-ansible-service-broker.* and other indices that can be selected from the left part.

If there are many projects, such as AA, BB, CC.. if we only add project.* index, we only can find project.* index in the left part of Kibana UI. We can not know the project name from the index. If we add project.AA*, project.BB*, .. indices to Kibana, it will make customers disappointed if they have many projects.

(In reply to Junqi Zhao from comment #26)
> (In reply to Rich Megginson from comment #25)
> > (In reply to Junqi Zhao from comment #24)
> > > (In reply to Junqi Zhao from comment #23)
> > > > @rmeggins
> > > > user can see the project indices by using workaround
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=1511432#c15
> > > > 
> > > > But if there are a lot of projects, it will make customers disappointed if
> > > > we let customers do the workaround manually
> > > 
> > > I mean if there are many projects, such as AA, BB, CC.., customers have to
> > > add project.AA*, project.BB*, .. indices to Kibana, if we jusut add
> > > project.* to Kibana, customers can not find every projects in Kibana
> > 
> > I'm not sure what you mean.
> 
> See the attached picture " Snapshot 1", we have .all, project.logging.*,
> project.openshift-ansible-service-broker.* and other indices that can be
> selected from the left part.

Right.  And that works fine when there are a very small number of projects.

> 
> If there are many projects, such as AA, BB, CC.. if we only add project.*
> index, we only can find project.* index in the left part of Kibana UI.

But that is not the only way to find projects.

> We can not know the project name from the index.

Correct, but there are other ways to find data for projects.

> If we add project.AA*,
> project.BB*, .. indices to Kibana, it will make customers disappointed if
> they have many projects.

Correct.

Please change to ON_QA, project indices could be found in kibana UI now.

Images
logging-kibana/images/v3.7.14-5
logging-fluentd/images/v3.7.14-5
logging-elasticsearch/images/v3.7.14-5
logging-auth-proxy/images/v3.7.14-5
logging-curator/images/v3.7.14-5

# openshift version
openshift v3.7.14
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

Created attachment 1367755 [details]
project indices could be found on kibana UI.

set to VERIFIED as per Comment 28

Are we tracking this for a release soon? Customer has a production readiness deadline approaching, and considers this a blocker

(In reply to Steven Walter from comment #31)
> Are we tracking this for a release soon? Customer has a production readiness
> deadline approaching, and considers this a blocker

@smunilla - did the last 3.7.z image release include logging-elasticsearch/images/v3.7.14-5 or later?  If not, is there an errata open already?

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0636