Bug 1939855 - Updateservice pod should be re-deployed when update graphDataImage of updateservice object
Summary: Updateservice pod should be re-deployed when update graphDataImage of updates...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OpenShift Update Service
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.12.z
Assignee: Over the Air Updates
QA Contact: Yang Yang
Kathryn Alexander
URL:
Whiteboard:
: 2030619 (view as bug list)
Depends On:
Blocks: 2048825
TreeView+ depends on / blocked
 
Reported: 2021-03-17 08:55 UTC by liujia
Modified: 2023-03-09 11:30 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Cause: A new graph data image will not Consequence: Workaround (if any): Result:
Clone Of:
Environment:
Last Closed: 2023-03-09 11:30:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cincinnati-operator pull 150 0 None open Bug 1939855: controllers/updateservice_controller.go: handle init container changes 2022-10-13 19:09:12 UTC
Red Hat Product Errata RHEA-2023:1161 0 None None None 2023-03-09 11:30:56 UTC

Description liujia 2021-03-17 08:55:28 UTC
Description of problem (please be detailed as possible and provide log
snippests):
Update graphDataImage of updateservice.spec will not get updateservice pod redeployed.

# ./oc get updateservice -ojson|jq .items[].metadata.generation
2
# ./oc get updateservice -ojson|jq .items[].spec
{
  "foo": "bar",
  "graphDataImage": "jliu-46.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rh-osbs/cincinnati-graph-data-container:1.1",
  "releases": "jliu-46.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp-release",
  "replicas": 1
}
# ./oc get deployment test -oyaml|grep image:|grep graph
      - image: jliu-46.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rh-osbs/cincinnati-graph-data-container:1.0

# ./oc get po test-65df76bf76-x79t7 -oyaml|grep image:|grep graph
  - image: jliu-46.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rh-osbs/cincinnati-graph-data-container:1.0
    image: jliu-46.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rh-osbs/cincinnati-graph-data-container:1.0

Version of all relevant components (if applicable):
OSUS operator image: v4.6.0-6
Bundle image: v1.0-21
Operand image: v4.6.0-8

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?
Delete the updateservice instance and re-create a new instance

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
always

Can this issue reproduce from the UI?
yes

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Create updateservice instance with graph-data image v1.0
...
spec:
  foo: bar
  graphDataImage: >-
    jliu-46.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rh-osbs/cincinnati-graph-data-container:1.0
  releases: 'jliu-46.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp-release'
  replicas: 1
...
2. Push graph-data image v1.1 to local registry for updating channel.yaml 

3. Update updateservice resource from web-console or cli to point graphDataImage to v1.1

Actual results:
The updateservice pod will not be re-deployed.

Expected results:
The new osus pod should be re-created with the new image automatically.

Additional info:
It also happened in following scenario that correcting the old wrong repo/image specified in updateservice.

Comment 1 Lalatendu Mohanty 2021-03-18 20:44:58 UTC
@liujia With compared to our dev preview release is this a regression?

Comment 3 liujia 2021-03-19 01:07:45 UTC
(In reply to Lalatendu Mohanty from comment #1)
> @liujia With compared to our dev preview release is this a regression?

I'm not sure the result of our dev preview release. It's tested by ACM team.

Comment 4 liujia 2021-03-19 01:47:45 UTC
Due to previous bundle image build issues, qe's test was blocked a lot. Now it's fixed well in v1.0.21, so our test against operator and operand start actually in recent days. We encountered several issues during past two days, even some of issues are by accident. So I added "needtestcases" keywords too to remind qe for adding cases later. I don't think all the issues are new in this GA version and maybe also in tech preview version. But since previous version is for tech preview and out of OTA team's control, so QE don't have an extra test against previous version. This is the 1st version of osus as a product, QE will focus on current version more to help finding/tracking existing issues before release. But i don't think all of bugs should be fixed in the 1st released version.

Comment 5 Lalatendu Mohanty 2021-03-24 15:31:35 UTC
This is not blocker for the first release and we need to add this bug to known issues. Hence pushing this to 4.8. Also restarting OSUS will not cause any impact on the availability of the OpenShift. I am changing the sev to medium.

Comment 6 Lalatendu Mohanty 2021-03-24 15:38:22 UTC
@jiajliu Are you saying without the restart of OSUS pod the new information in the graph-data image is not getting reflected in the graph?

Comment 7 liujia 2021-03-25 00:58:33 UTC
Yes, the update of graph-data image in updateservice resource will not get osus deployment/pod updated, and no restart of pod either. So it still use the old graph-data image which  does not include latest node/edge.

The workaround is to delete the old updateservice instance, and re-create a new one with the new graph-data image.

Comment 8 W. Trevor King 2021-10-08 01:49:00 UTC
*** Bug 2009651 has been marked as a duplicate of this bug. ***

Comment 9 W. Trevor King 2021-12-09 18:43:26 UTC
*** Bug 2030619 has been marked as a duplicate of this bug. ***

Comment 10 W. Trevor King 2022-02-21 19:41:44 UTC
Moving back to NEW now that the issue that #133 fixed has been moved over to bug 2009651.

Comment 16 Jack Ottofaro 2022-10-12 16:05:11 UTC
(In reply to Himanshu from comment #15)
> @jack.ottofaro Seeking update on this.

This is in work. Adding in behavior such that updates to graphDataImage are handled the same as updates to other currently supported attributes was straightforward. The issue is that none of the updates appear to be getting handled correctly. Rather than the existing operand pod simply being restarted updates result in an additional operand being created which is not correct.

Comment 18 Jack Ottofaro 2022-10-13 19:17:27 UTC
(In reply to Himanshu from comment #17)

Nothing required from the CU but I do have an update. Believe the general update issue I mentioned is not a real issue but a result of faulty testing. In my test env my replicaset was never getting to the ready state so, since deployment strategy is RollingUpdate, it creates a new replicaset but never deletes the old one since the new one also never gets to the ready state. Bottom line, I have a fix for the original issue described in the bug I just need to do some more testing.

Comment 29 Yang Yang 2023-02-06 08:01:02 UTC
Verifying on cincinnati-container-v5.0.1-3, cincinnati-operator-container-v5.0.1-3 and cincinnati-operator-bundle-container-v5.0.1-1

1. Install OSUS using quay.io/openshifttest/graph-data:5.0.1 as graph data image
# oc get updateservice -ojson|jq .items[].spec
{
  "graphDataImage": "quay.io/openshifttest/graph-data:5.0.1",
  "releases": "quay.io/openshifttest/ocp4/openshift4-release-images",
  "replicas": 1
}

# oc get pod
NAME                                      READY   STATUS    RESTARTS   AGE
sample-856879d565-vrmrl                   2/2     Running   0          27m
updateservice-operator-85758c57bb-977v5   1/1     Running   0          33m

2. Update the graph data image to 5.0.1-1
# oc edit updateservice
Vim: Warning: Output is not to a terminal
updateservice.updateservice.operator.openshift.io/sample edited

# oc get updateservice -ojson|jq .items[].spec
{
  "graphDataImage": "quay.io/openshifttest/graph-data:5.0.1-1",
  "releases": "quay.io/openshifttest/ocp4/openshift4-release-images",
  "replicas": 1
}


# oc get pod
NAME                                      READY   STATUS    RESTARTS   AGE
sample-547b879c59-5r4zp                   2/2     Running   0          28s
updateservice-operator-85758c57bb-977v5   1/1     Running   0          37m

# oc get pod sample-547b879c59-5r4zp -ojson | jq .status.initContainerStatuses
[
  {
    "containerID": "cri-o://63833255bd03193aafb37918923fc9fde253cf3257b0d7d3c8968d34caa5fa66",
    "image": "quay.io/openshifttest/graph-data:5.0.1-1",
    "imageID": "quay.io/openshifttest/graph-data@sha256:8c17430eddfee52c9f64d9be0b18b46d2b3078038887925a0d2110df3f01a917",
    "lastState": {},
    "name": "graph-data",
    "ready": true,
    "restartCount": 0,
    "state": {
      "terminated": {
        "containerID": "cri-o://63833255bd03193aafb37918923fc9fde253cf3257b0d7d3c8968d34caa5fa66",
        "exitCode": 0,
        "finishedAt": "2023-02-06T07:40:40Z",
        "reason": "Completed",
        "startedAt": "2023-02-06T07:40:40Z"
      }
    }
  }
]

UpdateService pod is updated with new graph data image. It looks good to me. Moving it to verified state.

Comment 31 errata-xmlrpc 2023-03-09 11:30:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHEA: OSUS Enhancement Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:1161


Note You need to log in before you can comment on or make changes to this bug.