Description of problem:
While trying to mirror CNV's images to a private namespace on quay.io I'm getting " http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=ENHANCE_YOUR_CALM, debug="""
It only happens when I'm trying to mirror the entire product. When trimming mapping.txt to have only 2 images, everything works as expected.
I belive that a limit for amount of requests per seconds should be added to "oc image mirror".
Version-Release number of selected component (if applicable):
Client Version: 4.5.0-0.okd-2020-07-29-070316
Mirror a product with 184 images to quay.io
All the images listed in images.txt should be mirrored to quay.io.
The command I ran:
oc image mirror -v=6 --filter-by-os '.*' -f mapping.txt --max-per-registry 1
I'm adding a partial log of the command, after a while quay just keep blocking the requests so I stopped it.
Created attachment 1713169 [details]
output of oc image mirror
Created attachment 1713170 [details]
Sally sync with Oleg, iirc this was discussed during that other mirroring issue, it's definitely worthwhile to
add such limit to mapping.
Looking at this now and in the upcoming sprint.
Can it be confirmed that 429s are coming back from quay.io? That would be rate limiting kicking in.
My suggestion would be to include some form of backoff and retry when this is encountered.
ICSP topic is being deferred until 4.7.
When running locally, I do see HTTP 429 response "Too Many Requests" although I don't see that in the logs attached here. I'll work up a PR with a retry - but as Maciej noted might be in 4.7 (this is not ICSP, though, but still, deferred until 4.7)
Getting back to this, actually I do not see 429 "Too Many Requests" error, I only see the reported:
"http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=ENHANCE_YOUR_CALM, debug="""
Reporter noted "when trimming mapping.txt to have only 2 images, everything works as expected." I have found that somewhere around 20 images, not 2, "oc image mirror --filter-by-os '.*' -f mapping.txt --max-per-registry 1" fails w/ above error when mirroring to quay, but not when mirroring to another registry, such as localhost:5000.
Retries do nothing, AFAICT. Only thing I've tried that works to prevent this error is this: https://github.com/openshift/oc/pull/570
I'll reach out to quay folks to see if I'm missing something.
ENHANCE_YOUR_CALM RESOURCE_EXHAUSTED ...with additional error detail provided by runtime to indicate that the exhausted resource is bandwidth.
Quay would return 429 for rate limits. ENHANCE_YOUR_CALM I'm unfamiliar with.
Actively working on this with quay engineers this sprint - Issue seems specific to quay. Mirroring multiple (>20 mappings) to docker.io and/or a localhost registry succeeds.
Actively working on this, and @TomMcKay, can you take a look - I sent the reproducer via slack, also. Thanks
Created attachment 1725093 [details]
mapping file to pass to oc image mirror, substitute sallyom for your quay username
Moving this over to Quay, engineers are looking into this. Feel free to move back if necessary and/or if a workaround in oc is required.
Please file issues in jira, bugzilla is not used
This problem exists with "oc adm catalog mirror" too. I do not see the options mentioned in #c18 so solution is required.
*** Bug 1938970 has been marked as a duplicate of this bug. ***
Confirmed with latest oc , the issue has fixed: the images list will be spliced , and won't reproduce the issue now .
Client Version: 4.8.0-202103210459.p0-1054107
[root@localhost ~]# oc image mirror --filter-by-os '.*' -f /tmp/mapp.txt --max-per-registry 1 --insecure
sha256:6813db353929ed1cac36875c1f1b33eb5d442f791a39c36d4f0652d18de88daa -> openstack-aodh-api
sha256:820de587b795b78b1f878e43cce62ac61aa2a2e327d9e6f7941ab440c618b6bc -> openstack-aodh-notifier
sha256:842fe33d4497d4e9288b665b7fc3cf2dcbd5324b3a4ffc7c771025461837038e -> openstack-aodh-evaluator
sha256:9766c12304bb46b5be300353db7cbffc885d64407026effc11b912eac23b518e -> openstack-base
sha256:994850e8c285ad698386b4c65d007280ed6aa75404d920e86a978c6611e2b591 -> openstack-barbican-keystone-listener
sha256:ac28d59f515106ef088a8037ece506cb948c083ac59bfdac46420057d16488dc -> openstack-barbican-api
sha256:b251230a08d184f62b59b16b6eb9ad1b0594d522a63e8de58164d093e62bc69a -> openstack-aodh-base
sha256:cba7b442e24feb2c0cd04d43dd3a9996af31b6b47a7409cd6c794d47b3bb3a63 -> openstack-barbican-worker
sha256:d17a1af01a24c8ccb4bfd94a647eddb19860609a357eb0fe0b88bd4e5897c82e -> openstack-barbican-base
sha256:de565769e71c851707c611ee7f35f9cb5cfc1af42d4de226cb5ec636f20d299f -> openstack-aodh-listener
stats: shared=6 unique=17 size=301.9MiB ratio=0.22
*** Bug 1938298 has been marked as a duplicate of this bug. ***
Actually, there's a much better solution to this bug, if you hit the error with authorization headers too long (ENHANCE_YOUR_CALM, other) - the correct solution is to add the flag `--skip-multiple-scopes=true`
This flag is not widely understood or known, we'll add some text to the help menu to clarify this.
This PR will be reverted, and when it is, please re-verify using the correct fix, that is, adding '--skip-multiple-scopes=true' to the 'oc image mirror' command.
Edit to https://bugzilla.redhat.com/show_bug.cgi?id=1874106#c27 the `--skip-multiple-scopes=true` is the workaround. The solution will be to identify when the scope size triggers the authorization header is too long issue. We will identify what that limit is, and investigate what the best action is when the limit is reached (which may be to default to --skip-multiple-scopes=true).
However, the PR that merged https://github.com/openshift/oc/pull/761 will be reverted since this workaround already exists, and we'll either leave this bz open or open a new bz to track the permanent fix.
Regarding BZ 1938298 and comment 27 above, there is no `--skip-multiple-scopes=true` option to the `oc adm catalog mirror` command.
https://github.com/openshift/oc/pull/780 has not merged yet, so , change status.
Zhou can you verify if this problem is still a thing for oc adm catalog mirror command?
I still cannot find the workaround flag "--skip-multiple-scopes" for the latest "oc" client, as follows:
[root@preserve-olm-env client]# ./oc adm catalog mirror --skip-multiple-scopes=true quay.io/openshift-qe-optional-operators/ocp4-index:latest quay.io/olmqe
Error: unknown flag: --skip-multiple-scopes
See 'oc adm catalog mirror --help' for usage.
[root@preserve-olm-env client]# ./oc version -o yaml
[root@preserve-olm-env client]# ./oc adm catalog mirror --help
Mirrors the contents of a catalog into a registry.
This command will pull down an image containing a catalog database, extract it to disk, query it to find all of the
images used in the manifests, and then mirror them to a target registry.
By default, the database is extracted to a temporary directory, but can be saved locally via flags.
An image content source policy is written to a file that can be added to a cluster with access to the target registry.
This will configure the cluster to pull from the mirrors instead of the locations referenced in the operator manifests.
A mapping.txt file is also created that is compatible with "oc image mirror". This may be used to further customize the
mirroring configuration, but should not be needed in normal circumstances.
oc adm catalog mirror SRC DEST [flags]
# Mirror an operator-registry image and its contents to a registry
oc adm catalog mirror quay.io/my/image:latest myregistry.com
# Mirror an operator-registry image and its contents to a particular namespace in a registry
oc adm catalog mirror quay.io/my/image:latest myregistry.com/my-namespace
# Mirror to an airgapped registry by first mirroring to files
oc adm catalog mirror quay.io/my/image:latest file:///local/index
oc adm catalog mirror file:///local/index/my/image:latest my-airgapped-registry.com
# Configure a cluster to use a mirrored registry
oc apply -f manifests/imageContentSourcePolicy.yaml
# Edit the mirroring mappings and mirror with "oc image mirror" manually
oc adm catalog mirror --manifests-only quay.io/my/image:latest myregistry.com
oc image mirror -f manifests/mapping.txt
--dir='': The directory on disk that file:// images will be copied under.
--dry-run=false: Print the actions that would be taken and exit without writing to the destinations.
--from-dir='': The directory on disk that file:// images will be read from. Overrides --dir
--icsp-scope='repository': Scope of registry mirrors in imagecontentsourcepolicy file. Allowed values: repository,
registry. Defaults to: repository
--index-filter-by-os='': A regular expression to control which index image is picked when multiple variants are
available. Images will be passed as '<platform>/<architecture>[/<variant>]'. This does not apply to images referenced by
--insecure=false: Allow push and pull operations to registries to be made over HTTP
--manifests-only=false: Calculate the manifests required for mirroring, but do not actually mirror image content.
--max-components=2: The maximum number of path components allowed in a destination mapping. Example:
`quay.io/org/repo` has two path components.
--max-per-registry=4: Number of concurrent requests allowed per registry.
--path='': Specify an in-container to local path mapping for the database.
-a, --registry-config='': Path to your registry credentials (defaults to ~/.docker/config.json)
--skip-verification=false: Skip verifying the integrity of the retrieved content. This is not recommended, but may
be necessary when importing images from older image registries. Only bypass verification if the registry is known to be
--to-manifests='': Local path to store manifests.
(In reply to Dale Bewley from comment #30)
> Regarding BZ 1938298 and comment 27 above, there is no
> `--skip-multiple-scopes=true` option to the `oc adm catalog mirror` command.
This option needs to be backported to OCP 4.7 at the very least - 4.6 would be nice too. Without this fix, nobody can use QUAY to hold an offline registry for OCP.
change status to ASSIGNED based on comment 36, 40
With the oc client in Release 4.6.23 (EUS), the exact same error occurs when mirroring the operators using "oc image mirror -f mapping.txt" and a QUAY ON-Prem installation. This does not occur with a container registry that is not based on QUAY (i.e. docker.io/registry:latest)
# oc version
Client Version: 4.6.23
# quay release
# export HTTPS_PROXY="...suqid-proxy.."
# export NO_PROXY="quay.xyz.org"
# oc image mirror -f mapping.txt
registry.redhat.io/rhscl/mongodb-36-rhel7 sha256:a858833a9239708c0c07c8fdf95218065c0605e14950051b009f9ad263f43511 1.765KiB
registry.redhat.io/rhscl/mongodb-36-rhel7 sha256:47536c692c5ac6a2f527383e4b8dfd7b485b1191942156ca6596506e1a11bee5 6.3KiB
registry.redhat.io/rhscl/mongodb-36-rhel7 sha256:608083cad0129a0f9240e5dcd4ceb087cc5ff025012277fc28bd77108e11a9bd 6.886MiB
registry.redhat.io/rhscl/mongodb-36-rhel7 sha256:b77f42d650dc7d0d6fa21f8661f03957cfe70fcf92e48245d2a7cad7d795eb56 72.89MiB
registry.redhat.io/rhscl/mongodb-36-rhel7 sha256:9ba25fd1823732e977fba3f71477f6fe55d88b168dcf02a12e4e2fc0c8f5f973 96.98MiB
sha256:5cd49903c19bb2fd2004b9d70f0af0386c1ec784aa63e91ec13f7ee472230770 -> bf36e238
registry.access.redhat.com/ubi8/ubi-minimal sha256:aebb8c5568533b57ee3da86262f7bff81383a2a624b9f54b9da3418705009901 1.699KiB
registry.access.redhat.com/ubi8/ubi-minimal sha256:28095021e526ad1dd5a65e11dc0fe4b34999ec398dbc60743f4b121d6bc9fc81 4.184KiB
registry.access.redhat.com/ubi8/ubi-minimal sha256:0fd3b5213a9b4639d32bf2ef6a3d7cc9891c4d8b23639ff7ae99d66ecb490a70 51.85MiB
sha256:f19c5b5d417cad1452ced0d174bca363ac41554190406c9147488b58394e2c56 -> 85d7f5f0
stats: shared=24 unique=254 size=5.168GiB ratio=0.85
error: unable to retrieve source image quay.xyz.org/ocp-operators-46/redhat-operator-index-46-bamf by tag v4.6:
http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=ENHANCOUR_CALM, debug=""
error: an error occurred during planning
Does anyone here know when these changess will "flow" into the oc 4.6 client (EUS) ?
In the meantime i'll try my luck with the oc-client 4.8.x.
> Does anyone here know when these changess will "flow" into the oc 4.6 cl
Some of the changes might get into 4.6, but this particular one might not due to the size of the incoming changes.
we set --skip-multiple-scopes to true for oc adm catalog mirror, the remaining commands require
this flag to be passed explicitly. Moving to qa for verification.
Created attachment 1790144 [details]
oc adm catalog mirror
This problem should be already addressed and fixed in 4.8. For oc adm catalog mirror the flag skipping multiple scopes is hardcoded since --skip-multiple-scopes is not exposed there.
For oc image mirror users are advised to use --skip-multiple-scopes when hitting issues as described above. I'm moving this to verification by qa.
ON_QA is strange. Of the two linked PRs, 761 made a change, and 780 reverted it. So no code linked to this bug going out, right? 780 mentions --skip-multiple-scopes; looks like 'oc image mirror' has had that since the openshift/oc repo was created. Maybe this should just be closed NOTABUG? Otherwise this bug would get shipped with a 4.8.z errata based on code that's been in OpenShift 4 for ages, and that feels like it's adding confusion, not reducing confusion.
(In reply to W. Trevor King from comment #58)
> ON_QA is strange. Of the two linked PRs, 761 made a change, and 780
> reverted it. So no code linked to this bug going out, right? 780 mentions
> --skip-multiple-scopes; looks like 'oc image mirror' has had that since the
> openshift/oc repo was created. Maybe this should just be closed NOTABUG?
> Otherwise this bug would get shipped with a 4.8.z errata based on code
> that's been in OpenShift 4 for ages, and that feels like it's adding
> confusion, not reducing confusion.
Trevor, if you look closely through https://github.com/openshift/oc/pull/780 you'll
notice it has two commits:
1. reverts #761
2. sets skip-multiple-scopes to true for `oc adm catalog mirror` which doesn't expose this flag.
This aligns with what I wrote in #comment 56 :-)
Ah, in that case having this ON_QA makes sense, and a target of 4.8.0 for code that shipped in 4.8's GA makes sense, but a target of 4.8.0 for a bug that will get associated with a 4.8.z errata is confusing. Possibly not as confusing as a 4.8.z target for code that shipped in 4.8's GA. So I dunno; maybe the 4.8.0 target is our best option.
[root@localhost tmp]# oc version --client
Client Version: 4.8.0-202109080022.p0.git.a0c12be.assembly.stream-a0c12be
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (OpenShift Container Platform 4.8.12 bug fix update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
Bug 1970458 is considering a 4.7.z backport. If there's a 4.6.z backport, it will depend on bug 1970458, but bug 1970458 would first need to be VERIFIED or later. Because we don't want to fix something in 4.6.z before we've fixed it in 4.7.z.