Bug 1866380

Summary:	must-gather pull policy should be changed to "Always"
Product:	OpenShift Container Platform	Reporter:	Yossi Segev <ysegev>
Component:	oc	Assignee:	Sally <somalley>
Status:	CLOSED ERRATA	QA Contact:	RamaKasturi <knarra>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.5	CC:	aos-bugs, jokerman, knarra, mfojtik, ncredi, pkliczew, tnozicka, yturgema
Target Milestone:	---	Keywords:	TestBlockerForLayeredProduct
Target Release:	4.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-27 16:24:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1859972

Description Yossi Segev 2020-08-05 13:08:59 UTC

Description of problem:
must-gather pull-policy is "IfNotPresent". This sometimes ends in using a non-latest build of must-gather image, e.g.:
When running:
$ oc adm must-gather --image=registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v2.4.0 --dest-dir=/home/cnv-qe-jenkins/yossi/mg

Let's assume there's already an image of v2.4.0 in the cluster (which remained there from a previous run of must-gather), e.g. from build #42, hence v2.4.0-42.
If the build *in the image repo* advanced, and now the latest there is #43 (v2.4.0-43), then the "IfNotPresent" pull-policy would prevent pulling the latest build.


Version-Release number of selected component (if applicable):
$ oc version
Client Version: 4.5.0-rc.1
Server Version: 4.5.0-rc.5
Kubernetes Version: v1.18.3+f291db1


How reproducible:
Always, although it is not easy to catch, as a user. There should be a significant difference between the existing, outdated image in the cluster and the one in the repo.


Steps to Reproduce:
1. Run a simple must-gather action, e.g.:
$ oc adm must-gather --image=registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-cnv-must-gather-rhel8:v2.4.0 --dest-dir=/home/cnv-qe-jenkins/yossi/mg


Actual results:
If an older image if the same version (v2.4.0 in this example) exists in the cluster - no new image will be pulled from the repo, even if the image in the repo is a newer build of the same version.


Expected results:
Always use the latest version (and build) of an image.


Additional info:
This was found whne using a CNV (Openshift Virtualization) must-gather image, but was debugged and found to be an issue the stems from the OCP must-gather pull-policy (in bz#1859972).

Comment 1 Sally 2020-08-20 19:34:53 UTC

The linked PR should merge this sprint to resolve this.

Comment 4 RamaKasturi 2020-08-26 10:59:53 UTC

Verified bug in the payload below and i see that the imagepullpolicy has been changed to Always.

[ramakasturinarra@dhcp35-60 verification-tests]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-08-26-032807   True        False         4h6m    Cluster version is 4.6.0-0.nightly-2020-08-26-032807
[ramakasturinarra@dhcp35-60 verification-tests]$ oc version -o yaml
clientVersion:
  buildDate: "2020-08-26T05:14:57Z"
  compiler: gc
  gitCommit: b08e3b04f1b19d9a7265b4c44372b44090262a87
  gitTreeState: clean
  gitVersion: 4.6.0-202008260439.p0-b08e3b0
  goVersion: go1.14.4
  major: ""
  minor: ""
  platform: linux/amd64
openshiftVersion: 4.6.0-0.nightly-2020-08-26-032807
serverVersion:
  buildDate: "2020-08-25T13:43:16Z"
  compiler: gc
  gitCommit: aaf4ce1cf2d4201a23e332030ccfeab6c17351fd
  gitTreeState: dirty
  gitVersion: v1.19.0-rc.2+aaf4ce1-dirty
  goVersion: go1.14.4
  major: "1"
  minor: 19+
  platform: linux/amd64


[ramakasturinarra@dhcp35-60 verification-tests]$ oc get pod must-gather-sm42d -o yaml | grep imagePullPolicy
            f:imagePullPolicy: {}
            f:imagePullPolicy: {}
    imagePullPolicy: Always
    imagePullPolicy: Always

Based on the above moving the bug to verified state

Comment 5 Yuval Turgeman 2020-09-14 06:05:01 UTC

Looks like the fix for this was reverted - https://github.com/openshift/oc/commit/de7c254f96746682bd7bbf7006a1f4da1751c114

Comment 6 Yossi Segev 2020-09-14 06:52:08 UTC

Good catch! Thanks, Yuval.
I submitted a needInfo question in the bug that is the source of this fix (https://bugzilla.redhat.com/show_bug.cgi?id=1874856#c5), with an ask to revert this.
Let's wait and see how that turns out.

Comment 7 Yuval Turgeman 2020-09-14 08:18:26 UTC

Well, you can look at the discussion on the PR https://github.com/openshift/oc/pull/549
I guess this one will be resolved as WONTFIX :/

Comment 13 errata-xmlrpc 2020-10-27 16:24:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196