Bug 1826781

Summary: Failed to upgrade logging from 4.4 to 4.5 - Could not resolve placeholder 'POD_IP'
Product: OpenShift Container Platform Reporter: IgorKarpukhin <ikarpukh>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Qiaoling Tang <qitang>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.5CC: anli, aos-bugs, bparees, mburke, openshift-bugzilla-robot, qitang
Target Milestone: ---Keywords: Reopened
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: EO operator wasn't updating ENV for deployment and stateful set Consequence: EO operator wasn't starting after 4.4 -> 4.5 upgrade Fix: Fixed by explicitly updating ENV variables for the EO operator Result: Bug fixed
Story Points: ---
Clone Of: 1809498
: 1829064 (view as bug list) Environment:
Last Closed: 2020-07-13 17:30:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1824006, 1829971    

Description IgorKarpukhin 2020-04-22 14:32:53 UTC
+++ This bug was initially created as a clone of Bug #1809498 +++

We are likely still awaiting proper images from ART since this is trying to boot an ES5 node.  ES6 will be used in 4.5.  What images does the EO say it should be using?  What images does the CLO say it should be using?

--- Additional comment from Anping Li on 2020-03-10 03:04:39 UTC ---

Yes, there isn't 4.5 channel in operator-registry redhat-operatrors-art. Waiting art.

--- Additional comment from Jeff Cantrill on 2020-04-14 20:08:20 UTC ---

Please retest now there are available images for 4.5. One suspicion I have is if you upgrade CLO before EO that records will be written to an index with a '-write' suffix and thus the new indexmanagement will be broken

Comment 3 Qiaoling Tang 2020-04-23 01:25:38 UTC
I tested upgrade from 4.4 to 4.5 again, still hit the issue in https://bugzilla.redhat.com/show_bug.cgi?id=1809498#c0. 

For the issue in https://bugzilla.redhat.com/show_bug.cgi?id=1809498#c4, it didn't appear. I checked the logging 4.4, the env var `POD_IP` was added to the elasticsearch when deploying logging 4.4.

I also tested upgrade from 4.3 to 4.5, the issue in https://bugzilla.redhat.com/show_bug.cgi?id=1809498#c4 didn't appear, but I hit the same issue in https://bugzilla.redhat.com/show_bug.cgi?id=1809498#c0. The env var `POD_IP` wasn't added to the elasticsearch when deploying logging 4.3.


It seems the fix PR https://github.com/openshift/elasticsearch-operator/pull/316 only fixes the issue in https://bugzilla.redhat.com/show_bug.cgi?id=1809498#c4. @IgorKarpukhin, is my understanding correct?


4.5 logging images are from 4.5.0-0.ci-2020-04-22-212726.

Comment 4 IgorKarpukhin 2020-04-23 11:23:58 UTC
Yes, that's correct. This fix isn't backported to 4.4 yet. I'm working on it.

Comment 5 IgorKarpukhin 2020-04-23 11:54:32 UTC
Qiaoling Tang could you please post the logs from catalog operator during the upgrade from 4.3 to 4.4?

Ex: oc logs -f catalog-operator-7648d9cff4-rqx9j -n openshift-operator-lifecycle-manager

Comment 6 Qiaoling Tang 2020-04-24 00:36:27 UTC
Catalog operator logs: http://pastebin.test.redhat.com/858655

Cluster version: 4.4.0-0.nightly-2020-04-23-192521

Comment 8 Jeff Cantrill 2020-04-28 18:06:54 UTC
Moving to ON_QA.  Qiaoling if you feel it is reasonable to move this bug to Verified please do so.,

Comment 9 Qiaoling Tang 2020-04-29 00:53:19 UTC
Per the discussions in the above, I move this to VERIFIED.

Feel free to reopen this bz if you have any problems related to this issue.

Comment 10 Jeff Cantrill 2020-05-07 17:13:34 UTC
*** Bug 1829971 has been marked as a duplicate of this bug. ***

Comment 11 IgorKarpukhin 2020-05-19 08:11:45 UTC
This issue has been fixed by this PR: https://github.com/openshift/elasticsearch-operator/pull/316. Not a bug anymore.

Comment 12 Ben Parees 2020-05-19 13:03:11 UTC
moving back to verified.  the release process will close this when appropriate.

Comment 13 errata-xmlrpc 2020-07-13 17:30:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409