Bug 1823606

Summary: The CLO doesn't update the Kibana when the cluster proxy policy changed.
Product: OpenShift Container Platform Reporter: Qiaoling Tang <qitang>
Component: LoggingAssignee: ewolinet
Status: CLOSED ERRATA QA Contact: Qiaoling Tang <qitang>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.4CC: aos-bugs, ewolinet, ikarpukh, jcantril, mburke
Target Milestone: ---   
Target Release: 4.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1826793 (view as bug list) Environment:
Last Closed: 2020-07-14 01:43:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1813209, 1826793    

Description Qiaoling Tang 2020-04-14 02:18:53 UTC
Description of problem:
The Kibana couldn't be updated when the proxy policy changed.

Version-Release number of selected component (if applicable):
clusterlogging.4.4.0-202004091036

How reproducible:
Always

Steps to Reproduce:
1. deploy logging
2. make some changes to the proxy policy, for example, add `spec.noProxy: test.no-proxy.com`

$ oc get proxy cluster -oyaml
apiVersion: config.openshift.io/v1
kind: Proxy
metadata:
  creationTimestamp: "2020-04-14T00:52:46Z"
  generation: 2
  name: cluster
  resourceVersion: "44634"
  selfLink: /apis/config.openshift.io/v1/proxies/cluster
  uid: 695a33ba-512e-48ca-89d1-7f535e078ffe
spec:
  noProxy: test.no-proxy.com
  trustedCA:
    name: ""

3. wait until all nodes restarted, check the Env Var in the deploy/cluster-logging-operator, deploy/kibana, daemonset/fluentd, the deploy/kibana isn't updated.


Actual results:
The deploy/kibana isn't updated after the proxy policy changed

Expected results:
The deploy/kibana could be updated

Additional info:
Workaround: manually delete the deploy/kibana, then wait CLO to create a new deploy/kibana

Comment 6 Qiaoling Tang 2020-04-24 02:48:45 UTC
Verified with images from 4.5.0-0.ci-2020-04-24-000603

Comment 7 Qiaoling Tang 2020-04-24 06:00:16 UTC
(In reply to Qiaoling Tang from comment #6)
> Verified with images from 4.5.0-0.ci-2020-04-24-000603
The testing cluster version was: 4.4.0-0.nightly-2020-04-23-192521  


I have to correct my testing result:

I used images from 4.5.0-0.ci-2020-04-24-012839, I tested in a 4.4 cluster and 4.5 cluster, the result are different:

Cluster version: 4.4.0-0.nightly-2020-04-23-192521:
Result: after the proxy/cluster changed, the deploy/kibana could be updated with correct proxy settings, the issue didn't appear.

Cluster version: 4.5.0-0.nightly-2020-04-21-103613:
Result: after the proxy/cluster changed, the deploy/kibana couldn't be updated with correct proxy settings, the issue in comment 0 appeared.

Comment 8 ewolinet 2020-04-24 14:59:15 UTC
@Qiaoling, 

Can you provide what your proxy/cluster object looked like for the 4.5 cluster that you were testing with?

From a logging perspective there should be no difference if you are using the latest 4.5 Elasticsearch-Operator image...
(the 4.4 fix for this will be in Cluster-Logging-Operator)

Comment 9 Qiaoling Tang 2020-04-26 01:11:04 UTC
Seems there is no difference between 4.4 and 4.5:

$ oc get proxy -o yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: Proxy
  metadata:
    creationTimestamp: "2020-04-26T00:09:41Z"
    generation: 1
    name: cluster
    resourceVersion: "804"
    selfLink: /apis/config.openshift.io/v1/proxies/cluster
    uid: 82b5b02d-efc6-4b72-ac8c-a6241a379165
  spec:
    trustedCA:
      name: ""
  status: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

cluster version:  4.5.0-0.nightly-2020-04-25-170442

Comment 10 ewolinet 2020-04-27 14:51:10 UTC
Since the `status` field is empty, I wouldn't expect the operator to make any changes... it makes ENV var updates based on the status field

Comment 11 Qiaoling Tang 2020-04-28 01:00:37 UTC
Sorry, I think I misunderstood you. The status field was updated after I added `noProxy: test.no-proxy.com` to the spec field, and I could see the ds/fluentd, deployment/cluster-logging-operator, deployment/elasticsearch-operator were updated. The proxy details I added in the above was an example of the proxy in the OCP 4.5, there didn't have any proxy settings in the spec field.

https://privatebin-it-iso.int.open.paas.redhat.com/?6a07209272505ed8#fNNMO2aQsR5YMN9EgpT00qRvz9LeQUym16Moo5V/LO4=

Comment 12 Qiaoling Tang 2020-04-28 01:02:46 UTC
(In reply to Qiaoling Tang from comment #11)
> Sorry, I think I misunderstood you. The status field was updated after I
> added `noProxy: test.no-proxy.com` to the spec field, and I could see the
> ds/fluentd, deployment/cluster-logging-operator,
> deployment/elasticsearch-operator were updated. The proxy details I added in
> the above was an example of the proxy in the OCP 4.5, there didn't have any
> proxy settings in the spec field.
> 
> https://privatebin-it-iso.int.open.paas.redhat.com/
> ?6a07209272505ed8#fNNMO2aQsR5YMN9EgpT00qRvz9LeQUym16Moo5V/LO4=

I pushed all the details in: `https://privatebin-it-iso.int.open.paas.redhat.com/?6a07209272505ed8#fNNMO2aQsR5YMN9EgpT00qRvz9LeQUym16Moo5V/LO4=`

Comment 13 ewolinet 2020-04-28 15:19:23 UTC
@Qiaoling,

I cannot view the details of that privatebin... 

> The status field was updated after I added `noProxy: test.no-proxy.com` to the spec field, and I could see the ds/fluentd, deployment/cluster-logging-operator, deployment/elasticsearch-operator were updated. The proxy details I added in the above was an example of the proxy in the OCP 4.5, there didn't have any proxy settings in the spec field.

I'm confused... so is this still an issue? It sounds like with the proxy settings we see this resolved and without any we don't see any ENV vars which is as expected.

Comment 15 ewolinet 2020-05-06 16:00:49 UTC
That's strange... can you please provide the logs for the Elasticsearch-Operator where you see this failing?

Comment 18 Jeff Cantrill 2020-05-11 19:22:22 UTC
@Qiaoling please set to VERIFIED per #c17 if appropriate or otherwise set the status.

Comment 19 Qiaoling Tang 2020-05-12 00:50:16 UTC
Tested with 4.5.0-0.nightly-2020-05-11-211039 and images from 4.5.0-0.ci-2020-05-11-212141, couldn't reproduce this issue. Move this bug to Verified.

Comment 25 errata-xmlrpc 2020-07-14 01:43:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2871