Bug 2088483

Summary: oc adm catalog mirror returns 0 even if there are errors
Product: OpenShift Container Platform Reporter: Alok Singh <alosingh>
Component: ocAssignee: Tyler Slaton <tyslaton>
oc sub component: oc QA Contact: Jian Zhang <jiazha>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: cgaynor, helwazer, jpower, juha.kosonen, mfojtik, tyslaton, vdinh
Version: 4.9   
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: All   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Errors were printed to console, but were not considered a failure. Consequence: The command always finished successfully, ie. zero exit code. Fix: Expose --continue-on-error to allow users to decide how the errors are being treated. Result: The errors can be fatal and cause the command to exit with a non-zero code.
 Story Points: ---
Clone Of:
: 2095584 (view as bug list) Environment:
Last Closed: 2022-08-10 11:13:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2095302, 2095584    

Description Alok Singh 2022-05-19 14:26:51 UTC 
Description of problem:

When catalog mirroring is done using 'oc adm catalog mirror', the command continues even if there are any errors and hence the return code of the command is 0. An equivalent of '--continue-on-error=false' of 'oc image mirror' command is required in 'oc adm catalog mirror' so that it would be possible to stop the execution when an error occurs and it would provide an error code.

The customer is planning to automate the operator catalog mirroring. Since the return code of the command is 0 even when there are some errors in mirroring, they have a blocker in implementing it as some images would still be missing after completion of the command execution.


Version-Release number of selected component (if applicable): 4.9.25


How reproducible:
Very

Steps to Reproduce:
1. Try mirroring with oc adm catalog mirror
2. The command continues even if an error occurs
3. The command completes execution with return code 0

Actual results:
The oc adm catalog mirror command continues even if there are errors and returns 0 after execution.

Expected results:
The command should have an option to stop execution when an error occurs similar to '--continue-on-error=false' of 'oc image mirror' command.


Additional info:

From Rui Gouveia(rgouveia):

"Source Code: https://github.com/openshift/oc

Executive summary:

The command "oc image mirror" has a parameter "--continue-on-error=false" that provides the control the customer wants. The command "oc adm catalog mirror" does not have this parameter, and in the code the later invokes the former hard-coding the missing options:

pkg/cli/admin/catalog/mirror.go
~~~
...
a := imgmirror.NewMirrorImageOptions(o.IOStreams)
a.SkipMissing = true
a.ContinueOnError = true <== hard-coded error handling parameter.
a.DryRun = o.DryRun
a.SecurityOptions = o.SecurityOptions
a.FilterOptions = allmanifests
a.ParallelOptions = o.ParallelOptions
a.KeepManifestList = true
a.Mappings = mappings
a.SkipMultipleScopes = true
if err := a.Validate(); err != nil {
fmt.Fprintf(o.IOStreams.ErrOut, "error configuring image mirroring: %v\n", err)
}
if err := a.Run(); err != nil {
fmt.Fprintf(o.IOStreams.ErrOut, "error mirroring image: %v\n", err)
}
return nil
...
~~~

In conclusion, as it is, in case of an error, the command continues without aborting.

I don't see a solution for this customer case, except to open a RFE asking for this parameter also to be implemented in the "oc image mirror". This way the code above could change to:

~~~
...
a.ContinueOnError = o.ContinueOnError
...
~~~

giving the customer the control of the execution flow."
Comment 1 Vu Dinh 2022-05-19 14:30:38 UTC 
Hi,

This is not oc-mirror issue. This is "oc adm catalog mirror" issue. Please file it against "oc" component. Redirect to oc component.

Thanks,
Vu
Comment 5 Colum Gaynor 2022-06-09 04:34:59 UTC 
@alosingh @helwazer -----> Please Create a Clone BZ to get the OCP 4.11 Fix Backported to OCP 4.10

Colum Gaynor - Senior Partner Success Manager, Nokia Global Account
Comment 6 Jian Zhang 2022-06-13 02:08:55 UTC 
1, Install the `oc` with the fixed PR.
[cloud-user@preserve-olm-env2 client]$ ./oc version
Client Version: 4.11.0-0.nightly-2022-06-11-120123
Kustomize Version: v4.5.4
Server Version: 4.11.0-0.nightly-2022-06-06-201913
Kubernetes Version: v1.24.0+bb9c2f1
[cloud-user@preserve-olm-env2 client]$ ./oc version -o yaml
clientVersion:
  buildDate: "2022-06-09T01:19:08Z"
  compiler: gc
  gitCommit: 194e99eb861a9bae2114d71f5459fb057d0ed36c
  gitTreeState: clean
  gitVersion: 4.11.0-202206090038.p0.g194e99e.assembly.stream-194e99e
  goVersion: go1.18
  major: ""
  minor: ""
  platform: linux/amd64
kustomizeVersion: v4.5.4
openshiftVersion: 4.11.0-0.nightly-2022-06-06-201913
...

2, check if the `--continue-on-error` option enabled.
[cloud-user@preserve-olm-env2 client]$ ./oc adm catalog mirror --help|grep continue -A1
    --continue-on-error=true:
	If an error occurs while mirroring, keep going and attempt to mirror as much as possible.

3, mirror a wrong image without the `--continue-on-error=false`.
[cloud-user@preserve-olm-env2 client]$ ./oc adm catalog mirror quay.io/olmqe/etcd-index:v3 localhost:5000 
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version of oc. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
src image has index label for declarative configs path: /configs/
using index path mapping: /configs/:/tmp/2101229420
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version of oc. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
wrote declarative configs to /tmp/2101229420
using declarative configs at: /tmp/2101229420
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version of oc. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
localhost:5000/
  olmqe/etcd-index
    blobs:
      quay.io/olmqe/etcd-index sha256:f254a54481e419dc0f67364fca0f78289f0a52490e96d62255afdfad394d70cd 159B
      quay.io/olmqe/etcd-index sha256:af64ecfdcb9496edc7b7badbe3028973175f071b6c31aa691fdf11aebf532402 394B
      quay.io/olmqe/etcd-index sha256:20c97716667af926e52cf0b62daa5bee4f8b98fe243bfc1b381fc9eba131ca1e 2.252KiB
      quay.io/olmqe/etcd-index sha256:c66c683f6df7971589dd1dd680dadbb8f04f3d993bbaaf355d73242730d2038a 2.321MiB
      quay.io/olmqe/etcd-index sha256:29291e31a76a7e560b9b7ad3cada56e8c18d50a96cca8a2573e4f4689d7aca77 2.683MiB
      quay.io/olmqe/etcd-index sha256:c4b1af403512539442e7af9cdc9918a210ecbb2138c617072da0994fc395d95a 3.602MiB
      quay.io/olmqe/etcd-index sha256:8a7cdc1b5153967d489fc2450a6f37d6a5d94a46c0f79863eb68fc3602008227 15.41MiB
    manifests:
      sha256:0258116849868880fd562a68c04ba7c3e7de9a0ad4476e332f3c0beb5fd75447 -> v3
  openshifttest/etcd-bundle
    blobs:
      quay.io/openshifttest/etcd-bundle sha256:d3a8e42f8e0dfb54d66dc78deeb0c1df7ee3fc8002545ac990476cce610096f0 283B
      quay.io/openshifttest/etcd-bundle sha256:22674ebc2791ed72bbef952fe9ba25e6a5c2953c96022f2435ce9c4ee2f2581c 2.114KiB
      quay.io/openshifttest/etcd-bundle sha256:e053c7d1c446c0e986106082eaddd8798e85aaaa6114b85df618528cc41d0e8d 8.52KiB
    manifests:
      sha256:676a4bd8c57c30615be5778a3dec9dc2e78e74f774a456de4e0a71ae882e269d -> 0.9.4
  stats: shared=0 unique=10 size=24.03MiB ratio=1.00

phase 0:
  localhost:5000 openshifttest/etcd-bundle blobs=3 mounts=0 manifests=1 shared=0
  localhost:5000 olmqe/etcd-index          blobs=7 mounts=0 manifests=1 shared=0

info: Planning completed in 830ms
error: unable to upload blob sha256:8a7cdc1b5153967d489fc2450a6f37d6a5d94a46c0f79863eb68fc3602008227 to localhost:5000/olmqe/etcd-index: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to upload blob sha256:c4b1af403512539442e7af9cdc9918a210ecbb2138c617072da0994fc395d95a to localhost:5000/olmqe/etcd-index: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/olmqe/etcd-index: failed to upload blob sha256:af64ecfdcb9496edc7b7badbe3028973175f071b6c31aa691fdf11aebf532402: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/olmqe/etcd-index: failed to upload blob sha256:20c97716667af926e52cf0b62daa5bee4f8b98fe243bfc1b381fc9eba131ca1e: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to upload blob sha256:29291e31a76a7e560b9b7ad3cada56e8c18d50a96cca8a2573e4f4689d7aca77 to localhost:5000/olmqe/etcd-index: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to upload blob sha256:c66c683f6df7971589dd1dd680dadbb8f04f3d993bbaaf355d73242730d2038a to localhost:5000/olmqe/etcd-index: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/olmqe/etcd-index: failed to upload blob sha256:f254a54481e419dc0f67364fca0f78289f0a52490e96d62255afdfad394d70cd: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push manifest to localhost:5000/olmqe/etcd-index:v3: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/openshifttest/etcd-bundle: failed to upload blob sha256:d3a8e42f8e0dfb54d66dc78deeb0c1df7ee3fc8002545ac990476cce610096f0: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/openshifttest/etcd-bundle: failed to upload blob sha256:e053c7d1c446c0e986106082eaddd8798e85aaaa6114b85df618528cc41d0e8d: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/openshifttest/etcd-bundle: failed to upload blob sha256:22674ebc2791ed72bbef952fe9ba25e6a5c2953c96022f2435ce9c4ee2f2581c: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push manifest to localhost:5000/openshifttest/etcd-bundle:0.9.4: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
info: Mirroring completed in 470ms (0B/s)
errors during mirroring. the full contents of the catalog may not have been mirrored: mirroring failed: error mirroring image: one or more errors occurred
no digest mapping available for quay.io/openshifttest/etcd-bundle:0.9.4, skip writing to ImageContentSourcePolicy
no digest mapping available for quay.io/olmqe/etcd-index:v3, skip writing to ImageContentSourcePolicy
wrote mirroring manifests to manifests-etcd-index-1655085479
deleted dir /tmp/2101229420

The status return code is 0.
[cloud-user@preserve-olm-env2 client]$ echo $?
0

4, mirror the wrong image with the `--continue-on-error=false` option.

[cloud-user@preserve-olm-env2 client]$ ./oc adm catalog mirror quay.io/olmqe/etcd-index:v3 localhost:5000 --continue-on-error=false
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version of oc. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
src image has index label for declarative configs path: /configs/
using index path mapping: /configs/:/tmp/662114843
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version of oc. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
wrote declarative configs to /tmp/662114843
using declarative configs at: /tmp/662114843
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version of oc. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
localhost:5000/
  olmqe/etcd-index
    blobs:
      quay.io/olmqe/etcd-index sha256:f254a54481e419dc0f67364fca0f78289f0a52490e96d62255afdfad394d70cd 159B
      quay.io/olmqe/etcd-index sha256:af64ecfdcb9496edc7b7badbe3028973175f071b6c31aa691fdf11aebf532402 394B
      quay.io/olmqe/etcd-index sha256:20c97716667af926e52cf0b62daa5bee4f8b98fe243bfc1b381fc9eba131ca1e 2.252KiB
      quay.io/olmqe/etcd-index sha256:c66c683f6df7971589dd1dd680dadbb8f04f3d993bbaaf355d73242730d2038a 2.321MiB
      quay.io/olmqe/etcd-index sha256:29291e31a76a7e560b9b7ad3cada56e8c18d50a96cca8a2573e4f4689d7aca77 2.683MiB
      quay.io/olmqe/etcd-index sha256:c4b1af403512539442e7af9cdc9918a210ecbb2138c617072da0994fc395d95a 3.602MiB
      quay.io/olmqe/etcd-index sha256:8a7cdc1b5153967d489fc2450a6f37d6a5d94a46c0f79863eb68fc3602008227 15.41MiB
    manifests:
      sha256:0258116849868880fd562a68c04ba7c3e7de9a0ad4476e332f3c0beb5fd75447 -> v3
  openshifttest/etcd-bundle
    blobs:
      quay.io/openshifttest/etcd-bundle sha256:d3a8e42f8e0dfb54d66dc78deeb0c1df7ee3fc8002545ac990476cce610096f0 283B
      quay.io/openshifttest/etcd-bundle sha256:22674ebc2791ed72bbef952fe9ba25e6a5c2953c96022f2435ce9c4ee2f2581c 2.114KiB
      quay.io/openshifttest/etcd-bundle sha256:e053c7d1c446c0e986106082eaddd8798e85aaaa6114b85df618528cc41d0e8d 8.52KiB
    manifests:
      sha256:676a4bd8c57c30615be5778a3dec9dc2e78e74f774a456de4e0a71ae882e269d -> 0.9.4
  stats: shared=0 unique=10 size=24.03MiB ratio=1.00

phase 0:
  localhost:5000 olmqe/etcd-index          blobs=7 mounts=0 manifests=1 shared=0
  localhost:5000 openshifttest/etcd-bundle blobs=3 mounts=0 manifests=1 shared=0

info: Planning completed in 790ms
error: unable to upload blob sha256:c66c683f6df7971589dd1dd680dadbb8f04f3d993bbaaf355d73242730d2038a to localhost:5000/olmqe/etcd-index: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to upload blob sha256:c4b1af403512539442e7af9cdc9918a210ecbb2138c617072da0994fc395d95a to localhost:5000/olmqe/etcd-index: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to upload blob sha256:8a7cdc1b5153967d489fc2450a6f37d6a5d94a46c0f79863eb68fc3602008227 to localhost:5000/olmqe/etcd-index: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to upload blob sha256:29291e31a76a7e560b9b7ad3cada56e8c18d50a96cca8a2573e4f4689d7aca77 to localhost:5000/olmqe/etcd-index: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/olmqe/etcd-index: failed to upload blob sha256:af64ecfdcb9496edc7b7badbe3028973175f071b6c31aa691fdf11aebf532402: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/olmqe/etcd-index: failed to upload blob sha256:20c97716667af926e52cf0b62daa5bee4f8b98fe243bfc1b381fc9eba131ca1e: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/olmqe/etcd-index: failed to upload blob sha256:f254a54481e419dc0f67364fca0f78289f0a52490e96d62255afdfad394d70cd: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/openshifttest/etcd-bundle: failed to upload blob sha256:22674ebc2791ed72bbef952fe9ba25e6a5c2953c96022f2435ce9c4ee2f2581c: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/openshifttest/etcd-bundle: failed to upload blob sha256:e053c7d1c446c0e986106082eaddd8798e85aaaa6114b85df618528cc41d0e8d: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
error: unable to push quay.io/openshifttest/etcd-bundle: failed to upload blob sha256:d3a8e42f8e0dfb54d66dc78deeb0c1df7ee3fc8002545ac990476cce610096f0: Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
info: Mirroring completed in 200ms (0B/s)
deleted dir /tmp/662114843
error: errors during mirroring. the full contents of the catalog may not have been mirrored: mirroring failed: error mirroring image: one or more errors occurred while uploading images

The status return code is 1.
[cloud-user@preserve-olm-env2 client]$ echo $?
1

LGTM, verify it.
Comment 7 Alok Singh 2022-06-14 13:59:55 UTC 
@cgaynor  The clone BZ was created for 4.10 - https://bugzilla.redhat.com/show_bug.cgi?id=2095584
Comment 10 errata-xmlrpc 2022-08-10 11:13:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069
Comment 12 Red Hat Bugzilla 2023-09-15 01:55:00 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days