Bug 1841588

Summary: CAM 1.2.0 release broke 1.1.1 SHAs
Product: OpenShift Container Platform Reporter: Xin jiang <xjiang>
Component: Migration ToolingAssignee: Jason Montleon <jmontleo>
Status: CLOSED ERRATA QA Contact: Xin jiang <xjiang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.4CC: chezhang, jmatthew, jmontleo, mberube, rjohnson, sregidor, whu
Target Milestone: ---   
Target Release: 4.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1841586 Environment:
Last Closed: 2020-06-04 11:35:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1841586    
Bug Blocks:    

Description Xin jiang 2020-05-29 14:07:22 UTC 
+++ This bug was initially created as a clone of Bug #1841586 +++

Description of problem:
CAM 1.2.0 release broke 1.1.1 SHAs, the some of 1.1.1 SHAs are not correct

Version-Release number of selected component (if applicable):
CAM 1.2.0

How reproducible:
Always

Steps to Reproduce:
While we are mirroring images through the latest CAM 1.2.0 release bundle, found some of 1.1.1 SHAs are not correct

Actual results:
Logs:
TASK [operator-mirror : Update images in cvs to point to the private registry] ***

changed: [localhost] => (item={'path': '/tmp/ansible.cm_izi0e-cam-operator_redhat_download/manifests/cam-operator/v1.1.1/konveyor-operator.v1.1.1.clusterserviceversion.yaml', 'regex': 'registry.redhat.io', 'replace': 'quay-enterprise-quay-enterprise.apps.cam-tgt-4380.qe.devcluster.openshift.com'})

changed: [localhost] => (item={'path': '/tmp/ansible.cm_izi0e-cam-operator_redhat_download/manifests/cam-operator/v1.1.1/konveyor-operator.v1.1.1.clusterserviceversion.yaml', 'regex': 'rhcam-1-1', 'replace': 'admin'})


TASK [operator-mirror : Get image tag] *****************************************

ok: [localhost]


TASK [operator-mirror : Mirror SHA images of the new csv_version from the specified repository] ***

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (5 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (4 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (3 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (2 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (1 retries left).

failed: [localhost] (item={'registry': 'registry.redhat.io', 'ns': 'rhcam-1-1', 'name': 'openshift-migration-controller-rhel8', 'sha': 'cce92d80929675fc14b6ad198375a7af4acbeef1f21c7c2e642ab0f13ba0d87e'}) => {"ansible_loop_var": "item", "attempts": 5, "changed": true, "cmd": "oc image mirror --insecure=true registry.redhat.io/rhcam-1-1/openshift-migration-controller-rhel8@sha256:cce92d80929675fc14b6ad198375a7af4acbeef1f21c7c2e642ab0f13ba0d87e                              quay-enterprise-quay-enterprise.apps.cam-tgt-4380.qe.devcluster.openshift.com/admin/openshift-migration-controller-rhel8:v1.1\n", "delta": "0:00:01.118579", "end": "2020-05-29 08:37:53.414486", "item": {"name": "openshift-migration-controller-rhel8", "ns": "rhcam-1-1", "registry": "registry.redhat.io", "sha": "cce92d80929675fc14b6ad198375a7af4acbeef1f21c7c2e642ab0f13ba0d87e"}, "msg": "non-zero return code", "rc": 1, "start": "2020-05-29 08:37:52.295907", "stderr": "\nerror: unable to retrieve source image registry.redhat.io/rhcam-1-1/openshift-migration-controller-rhel8 manifest sha256:cce92d80929675fc14b6ad198375a7af4acbeef1f21c7c2e642ab0f13ba0d87e: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: \"File not found.\\\"\"\nerror: an error occurred during planning", "stderr_lines": ["", "error: unable to retrieve source image registry.redhat.io/rhcam-1-1/openshift-migration-controller-rhel8 manifest sha256:cce92d80929675fc14b6ad198375a7af4acbeef1f21c7c2e642ab0f13ba0d87e: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: \"File not found.\\\"\"", "error: an error occurred during planning"], "stdout": "", "stdout_lines": []}

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (5 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (4 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (3 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (2 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (1 retries left).

failed: [localhost] (item={'registry': 'registry.redhat.io', 'ns': 'rhcam-1-1', 'name': 'openshift-migration-plugin-rhel8', 'sha': 'a0761f3014e50ba6832bccff248d53e1cb41bf8a3100833bc05adec26eb04c49'}) => {"ansible_loop_var": "item", "attempts": 5, "changed": true, "cmd": "oc image mirror --insecure=true registry.redhat.io/rhcam-1-1/openshift-migration-plugin-rhel8@sha256:a0761f3014e50ba6832bccff248d53e1cb41bf8a3100833bc05adec26eb04c49                              quay-enterprise-quay-enterprise.apps.cam-tgt-4380.qe.devcluster.openshift.com/admin/openshift-migration-plugin-rhel8:v1.1\n", "delta": "0:00:01.117528", "end": "2020-05-29 08:38:52.252228", "item": {"name": "openshift-migration-plugin-rhel8", "ns": "rhcam-1-1", "registry": "registry.redhat.io", "sha": "a0761f3014e50ba6832bccff248d53e1cb41bf8a3100833bc05adec26eb04c49"}, "msg": "non-zero return code", "rc": 1, "start": "2020-05-29 08:38:51.134700", "stderr": "\nerror: unable to retrieve source image registry.redhat.io/rhcam-1-1/openshift-migration-plugin-rhel8 manifest sha256:a0761f3014e50ba6832bccff248d53e1cb41bf8a3100833bc05adec26eb04c49: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: \"File not found.\\\"\"\nerror: an error occurred during planning", "stderr_lines": ["", "error: unable to retrieve source image registry.redhat.io/rhcam-1-1/openshift-migration-plugin-rhel8 manifest sha256:a0761f3014e50ba6832bccff248d53e1cb41bf8a3100833bc05adec26eb04c49: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: \"File not found.\\\"\"", "error: an error occurred during planning"], "stdout": "", "stdout_lines": []}

changed: [localhost] => (item={'registry': 'registry.redhat.io', 'ns': 'rhcam-1-1', 'name': 'openshift-migration-registry-rhel8', 'sha': '0ae610db4f73b6a5353c4821165bd60a8c4e86ac5ba5f1d60cd532f5bcd814bd'})

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (5 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (4 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (3 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (2 retries left).

FAILED - RETRYING: Mirror SHA images of the new csv_version from the specified repository (1 retries left).

failed: [localhost] (item={'registry': 'registry.redhat.io', 'ns': 'rhcam-1-1', 'name': 'openshift-migration-rhel7-operator', 'sha': 'da62d312c7d0f6e81e289192c6ce84953879341661dccd704058c3b150b806e7'}) => {"ansible_loop_var": "item", "attempts": 5, "changed": true, "cmd": "oc image mirror --insecure=true registry.redhat.io/rhcam-1-1/openshift-migration-rhel7-operator@sha256:da62d312c7d0f6e81e289192c6ce84953879341661dccd704058c3b150b806e7                              quay-enterprise-quay-enterprise.apps.cam-tgt-4380.qe.devcluster.openshift.com/admin/openshift-migration-rhel7-operator:v1.1\n", "delta": "0:00:01.305911", "end": "2020-05-29 08:39:53.505526", "item": {"name": "openshift-migration-rhel7-operator", "ns": "rhcam-1-1", "registry": "registry.redhat.io", "sha": "da62d312c7d0f6e81e289192c6ce84953879341661dccd704058c3b150b806e7"}, "msg": "non-zero return code", "rc": 1, "start": "2020-05-29 08:39:52.199615", "stderr": "\nerror: unable to retrieve source image registry.redhat.io/rhcam-1-1/openshift-migration-rhel7-operator manifest sha256:da62d312c7d0f6e81e289192c6ce84953879341661dccd704058c3b150b806e7: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: \"File not found.\\\"\"\nerror: an error occurred during planning", "stderr_lines": ["", "error: unable to retrieve source image registry.redhat.io/rhcam-1-1/openshift-migration-rhel7-operator manifest sha256:da62d312c7d0f6e81e289192c6ce84953879341661dccd704058c3b150b806e7: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: \"File not found.\\\"\"", "error: an error occurred during planning"], "stdout": "", "stdout_lines": []}

changed: [localhost] => (item={'registry': 'registry.redhat.io', 'ns': 'rhcam-1-1', 'name': 'openshift-migration-ui-rhel8', 'sha': '11c8c729bd4871c33b52f0405a0c5ec0438aefd2ed0ffc712034ba89160ed133'})

changed: [localhost] => (item={'registry': 'registry.redhat.io', 'ns': 'rhcam-1-1', 'name': 'openshift-migration-velero-plugin-for-aws-rhel8', 'sha': '460dfc455de7ee6a2e49d17d5227c5d653340197b7ad9ed430576c35f4651f4d'})

changed: [localhost] => (item={'registry': 'registry.redhat.io', 'ns': 'rhcam-1-1', 'name': 'openshift-migration-velero-plugin-for-gcp-rhel8', 'sha': '44f40ff5a3c8ad9b76105e2b8fc5bd04692464cc4aa683da2cf83b3336200863'})

changed: [localhost] => (item={'registry': 'registry.redhat.io', 'ns': 'rhcam-1-1', 'name': 'openshift-migration-velero-plugin-for-microsoft-azure-rhel8', 'sha': '9e69f2af712452218cde0c3325c60f9e1eb4624bcaff67770822b60f2e19ac60'})

changed: [localhost] => (item={'registry': 'registry.redhat.io', 'ns': 'rhcam-1-1', 'name': 'openshift-migration-velero-restic-restore-helper-rhel8', 'sha': '44f0362c8570d707582bd428aaf18f390ce915ef72cdeb60cf2699171dbda3c8'})

changed: [localhost] => (item={'registry': 'registry.redhat.io', 'ns': 'rhcam-1-1', 'name': 'openshift-migration-velero-rhel8', 'sha': 'db2997115d8a0767d61038e14f48170dc53d3c54b977184e7ecb37ead2f131da'})


Expected results:
They should be mirrored correctly

Additional info:
Comment 1 Jason Montleon 2020-05-29 14:10:10 UTC 
I'm honestly not sure what happened here and I'm guessing it might have been due to some churn caused by the konveyor rename, though even that's just a guess. We've reverted the changes so this should be fixed when the 1.2.1 errata goes out.
Comment 5 Sergio 2020-06-03 17:28:23 UTC 
Verified with CAM 1.2.1 stage

With 1.2.1 we can mirror all the images without problems.
Comment 7 errata-xmlrpc 2020-06-04 11:35:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2387