Bug 1575662 - Documents indexed under the wrong index when namespaces are deleted/recreated
Summary: Documents indexed under the wrong index when namespaces are deleted/recreated
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 3.11.z
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On: 1568894
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-07 14:51 UTC by Nicolas Nosenzo
Modified: 2021-03-01 08:28 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1568894
Environment:
Last Closed: 2020-05-12 00:41:12 UTC
Target Upstream Version:


Attachments (Terms of Use)

Comment 1 Nicolas Nosenzo 2018-05-07 15:08:57 UTC
Still failing in:

oc v3.9.14
kubernetes v1.9.1+a0ce1bc657
fluentd v3.9.14-2

Now checking with latest fluentd image for 3.9

Comment 2 Nicolas Nosenzo 2018-05-07 15:21:30 UTC
Fails also for logging-fluentd-v3.9.25-1

$ sh estest.sh 
1cfc5479-5207-11e8-925a-bab7b04c3d1f
^C
[quicklab@master-0 ~]$ sh estest1.sh 
1cfc5479-5207-11e8-925a-bab7b04c3d1f
4e2bb589-5208-11e8-925a-bab7b04c3d1f
5aa44177-5208-11e8-925a-bab7b04c3d1f
6896bdf0-5208-11e8-925a-bab7b04c3d1f
7a13bd50-5208-11e8-925a-bab7b04c3d1f
874460ce-5208-11e8-925a-bab7b04c3d1f
93be2569-5208-11e8-925a-bab7b04c3d1f
a1259a95-5208-11e8-925a-bab7b04c3d1f
ad6ce706-5208-11e8-925a-bab7b04c3d1f
bae6d13f-5208-11e8-925a-bab7b04c3d1f
c832f30d-5208-11e8-925a-bab7b04c3d1f
d5ba3457-5208-11e8-925a-bab7b04c3d1f
e3228722-5208-11e8-925a-bab7b04c3d1f
f00afe2e-5208-11e8-925a-bab7b04c3d1f
fdb7f12d-5208-11e8-925a-bab7b04c3d1f
0afcbdda-5209-11e8-925a-bab7b04c3d1f
17343b85-5209-11e8-925a-bab7b04c3d1f
23b27ca8-5209-11e8-925a-bab7b04c3d1f
2fdd3f1d-5209-11e8-925a-bab7b04c3d1f
3d22c2d5-5209-11e8-925a-bab7b04c3d1f


[quicklab@master-0 ~]$  oc rsh -c elasticsearch $es_pod es_util  --query=_cat/indices?h=index,docs.count | grep pelle
project.pelle-foo.1cfc5479-5207-11e8-925a-bab7b04c3d1f.2018.05.07                 1 


[quicklab@master-0 ~]$ oc get po -o 'go-template={{range $pod := .items}}{{range $container := $pod.spec.containers}}oc exec -c {{$container.name}} {{$pod.metadata.name}} -- find /root/buildinfo -name Dockerfile-openshift*{{"\n"}}{{end}}{{end}}' | bash -
/root/buildinfo/Dockerfile-openshift3-logging-curator-v3.9.14-2
/root/buildinfo/Dockerfile-openshift3-logging-elasticsearch-v3.9.14-2
/root/buildinfo/Dockerfile-openshift3-oauth-proxy-v3.9.14-2
/root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.25-1
/root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.25-1
/root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.9.25-1
/root/buildinfo/Dockerfile-openshift3-logging-kibana-v3.9.14-2
/root/buildinfo/Dockerfile-openshift3-logging-auth-proxy-v3.9.14-3

Comment 3 Jeff Cantrill 2018-05-11 21:05:43 UTC
Master PR to add test to CI

Comment 5 Rich Megginson 2018-05-16 02:07:27 UTC
In addition to https://github.com/openshift/origin-aggregated-logging/pull/1145 we need
- a hack/testing/test-indexing_after_ns_removal.sh which calls test/indexing_after_ns_removal.sh
- need to verify that the kubernetes metadata in the record matches the expected data e.g. the pod uuid in the record must match the actual pod uuid, the namespace uuid must match the actual namespace uuid and must match the namespace uuid in the index name
- the test must check to see that there are no orphaned records in the .orphaned index

Comment 6 Nicolas Nosenzo 2018-05-29 13:00:27 UTC
Master PR including above comments:
https://github.com/openshift/origin-aggregated-logging/pull/1190

Comment 14 Stephen Cuppett 2019-11-20 18:56:07 UTC
OCP 3.6-3.10 is no longer on full support [1]. Marking CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Target Release to the appropriate version where needed.

[1]: https://access.redhat.com/support/policy/updates/openshift

Comment 15 Nicolas Nosenzo 2019-11-25 12:24:27 UTC
Re-opening this BZ, since this is still happening in v3.11.153 EFK images.

Comment 16 Jeff Cantrill 2019-12-06 16:05:52 UTC
(In reply to Nicolas Nosenzo from comment #15)
> Re-opening this BZ, since this is still happening in v3.11.153 EFK images.

Can you provide examples of what you are seeing?  

* In what index are the logs found?
* What is the ns/nsuuid of the expected ns

Comment 17 Nicolas Nosenzo 2019-12-06 19:05:55 UTC
(In reply to Jeff Cantrill from comment #16)
> Can you provide examples of what you are seeing?  
> 
> * In what index are the logs found?
> * What is the ns/nsuuid of the expected ns

Just found my lab wiped out, but I will come back with those examples early next week. Just fyi, I was using the test suite `indexing_after_ns_removal.sh`


Note You need to log in before you can comment on or make changes to this bug.