Bug 2055460 - OUS uses wrong imagePullPolicy for graph-data initContainer
Summary: OUS uses wrong imagePullPolicy for graph-data initContainer
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OpenShift Update Service
Version: 4.6
Hardware: x86_64
OS: All
medium
high
Target Milestone: ---
: 4.9.z
Assignee: Over the Air Updates
QA Contact: liujia
Kathryn Alexander
URL:
Whiteboard:
Depends On: 2009651
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-02-17 02:49 UTC by Lalatendu Mohanty
Modified: 2022-06-28 06:40 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The update service deployment used pullIfNotPresent for the graph-data container. Consequence: When the configured UpdateService graphDataImage used a by-tag pullspec, and the host node contained a cached image for that pullspec, new update service pods might not notice updated graph data and could continue to serve old graph data. Fix: The graph-data container is now pullAlways. Result: New update service pods will always retrieve fresh graph data, even when graphDataImage is configured with a by-tag pullspec.
Clone Of: 2009651
Environment:
Last Closed: 2022-02-25 00:52:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cincinnati-operator pull 142 0 None Merged Bug 2048825: controllers/new: Always pull graph-data images 2022-02-17 02:52:25 UTC
Red Hat Product Errata RHBA-2022:0681 0 None None None 2022-02-25 00:53:00 UTC

Description Lalatendu Mohanty 2022-02-17 02:49:28 UTC
+++ This bug was initially created as a clone of Bug #2009651 +++

Description of problem (please be detailed as possible and provide log
snippests):

Current definition of a UpdateService deployment has "IfNotPresent" as a imagePullPolicy defined, which blocks updated graph-data images to be used. "Always" should be used for the initContainer.

      initContainers:
        - name: graph-data
          image: 'registry.example.com/ocp4/graph-data:latest'
          resources: {}
          volumeMounts:
            - name: cincinnati-graph-data
              mountPath: /var/lib/cincinnati/graph-data
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent      <== should be Always


Version of all relevant components (if applicable):

update-service-operator.v4.6.0
registry.redhat.io/openshift-update-service/openshift-update-service-rhel8-operator@sha256:08b4fc72501e5f7dfdd779e101d98b913d77982c021998f47b3cdc0367d7e0fa
registry.redhat.io/openshift-update-service/openshift-update-service-rhel8@sha256:9748a280f2a04524da739d2f8b7d8a74c5b58170c9c40b3e40904dd8ca39fbe8

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

The initial deployment of a update-service will pull an graph-data image tagged "latest" - but subsequent updates to that image will not get pulled as this tag already exists on a node. The only way to mitigate that effect is to use different tags for the graph-data images. This on the other hand requires additional adjustments made to the update-service CR.

Is there any workaround available to the best of your knowledge?
Changing graph-data image tag from "latest" to a incremental one.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
yes

Can this issue reproduce from the UI?
yes

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. install and deploy OUS according to the docs
2. create graph-data image as described in the docs
3. verify OUS functionality, rendering data of graph-data at that time.
4. whenever a update version of the update graph is available, update the graph-data image
5. OUS won't make use of updated graph-data image as <image>:latest is already on the node - even though it changed.


Actual results:
Outdated graph-data image will get used.

Expected results:
graph-data image is pulled every time to maintain current data.

Additional info:

--- Additional comment from Andreas Bleischwitz on 2021-10-01 07:47:25 UTC ---

If I'm not looking at the wrong point, the following needs to get changed:

https://github.com/openshift/cincinnati-operator/blob/f4aae3fdaaf8174f2159437f796b775e9b7efe52/controllers/new.go#L425

		Image:           instance.Spec.GraphDataImage,
		ImagePullPolicy: corev1.PullIfNotPresent,
		VolumeMounts: []corev1.VolumeMount{

to
		Image:           instance.Spec.GraphDataImage,
		ImagePullPolicy: corev1.Always,
		VolumeMounts: []corev1.VolumeMount{

Or remove the line containing "ImagePullPolicy:" completely as the default is:

// Defaults to Always if :latest tag is specified, or IfNotPresent otherwise.

Which probably is the best option for this image.

--- Additional comment from liujia on 2021-10-08 01:33:56 UTC ---

A known issue? Refer to https://bugzilla.redhat.com/show_bug.cgi?id=1939855.

--- Additional comment from W. Trevor King on 2021-10-08 01:49:00 UTC ---

hah, yes.  I'll close this one as a dup and attach my PR to the older issue.

--- Additional comment from Lalatendu Mohanty on 2022-02-17 02:46:31 UTC ---

As per comment in https://bugzilla.redhat.com/show_bug.cgi?id=2048825#c4 , looks like this bug is fixed by https://github.com/openshift/cincinnati-operator/pull/133

Comment 2 liujia 2022-02-17 09:19:13 UTC
Reproduced Version:
cincinnati-container-v4.9.0-3
cincinnati-operator-bundle-container-v4.9.0-7
cincinnati-operator-container-v4.9.0-3

steps:
1. Install osus operator v4.9.0 on ocp v4.9
2. Build graph-data image v1.0.0 and push to registry as cincinnati-graph-data-container:latest
3. Create updateservice instance with cincinnati-graph-data-container:latest
# ./oc get po sample-65bf4646b8-xvkcx -oyaml|grep cincinnati-graph-data-container
  - image: quay.io/openshifttest/cincinnati-graph-data-container:latest
    image: quay.io/openshifttest/cincinnati-graph-data-container:latest
    imageID: quay.io/openshifttest/cincinnati-graph-data-container@sha256:fc97cb8e57aba5048bd83a880d3ae0f9aea600b1e730be7f6379f4576aa1af2d
# ./oc get po sample-65bf4646b8-xvkcx -oyaml|grep -A3 initContainers:
  initContainers:
  - image: quay.io/openshifttest/cincinnati-graph-data-container:latest
    imagePullPolicy: IfNotPresent
    name: graph-data

4. Build graph-data image v1.0.1 and push to registry as cincinnati-graph-data-container:latest again.
5. Delete the old updateservice pod and re-create updaterservice with cincinnati-graph-data-container:latest
# ./oc get po sample-65bf4646b8-lkqwg -oyaml|grep cincinnati-graph-data-container
  - image: quay.io/openshifttest/cincinnati-graph-data-container:latest
    image: quay.io/openshifttest/cincinnati-graph-data-container:latest
    imageID: quay.io/openshifttest/cincinnati-graph-data-container@sha256:fc97cb8e57aba5048bd83a880d3ae0f9aea600b1e730be7f6379f4576aa1af2d

The graph-data image is still the old one @sha256:fc97cb8e57aba5048bd83a880d3ae0f9aea600b1e730be7f6379f4576aa1af2d.

Comment 3 liujia 2022-02-17 09:20:42 UTC
Verified Version:
cincinnati-container-v4.9.0-9
cincinnati-operator-bundle-container-v4.9.1-2
cincinnati-operator-container-v4.9.0-10

Before upgrade graph-data:latest image:
# ./oc get po sample-7d9dcddbbd-4vbcn -oyaml|grep cincinnati-graph-data-container
  - image: quay.io/openshifttest/cincinnati-graph-data-container:latest
    image: quay.io/openshifttest/cincinnati-graph-data-container:latest
    imageID: quay.io/openshifttest/cincinnati-graph-data-container@sha256:03c0aef42e61c736f470ad471c4709cfc93aa9acb2f06852a2cc502de491aa6c

After update graph-data image and re-create updateservice pod:
# ./oc get po sample-7d9dcddbbd-f4c9r -oyaml|grep cincinnati-graph-data-container
  - image: quay.io/openshifttest/cincinnati-graph-data-container:latest
    image: quay.io/openshifttest/cincinnati-graph-data-container:latest
    imageID: quay.io/openshifttest/cincinnati-graph-data-container@sha256:fc97cb8e57aba5048bd83a880d3ae0f9aea600b1e730be7f6379f4576aa1af2d

The imagePullPolicy was updated to Always:
# ./oc get po sample-7d9dcddbbd-4vbcn -oyaml|grep -A3 initContainers:
  initContainers:
  - image: quay.io/openshifttest/cincinnati-graph-data-container:latest
    imagePullPolicy: Always
    name: graph-data

Comment 5 errata-xmlrpc 2022-02-25 00:52:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHEA: OSUS enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0681

Comment 6 liujia 2022-06-28 06:40:22 UTC
case OCP-52586 added.


Note You need to log in before you can comment on or make changes to this bug.