Cause:
During mirroring we create an authorization header with all the image information.
Consequence:
Some registries have limits on header size which are violated by the size.
Fix:
Force --skip-multiple-scopes for oc adm catalog mirror command.
Result:
Mirroring works as expected.
Description of problem:
So when we are trying to mirror the image from the pruned image to private registry by running the below command :
oc adm catalog mirror \
xyz.azurecr.io/redhat/redhat-operator-index:v4.6 \
xyz.azurecr.io \
-a ${REG_CREDS} \
--filter-by-os='.*'
I get the below error for all the images that we are trying to push. below is just 1 sample.
error: unable to push manifest to xyz.azurecr.io/rhel8/skopeo:9fc07dfb: Put "https://xyz.azurecr.io/v2/rhel8/skopeo/manifests/9fc07dfb": error parsing HTTP 414 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>414 Request-URI Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>414 Request-URI Too Large</h1></center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n"
Version-Release number of selected component (if applicable):
- OCP 4.6
How reproducible:
- everytime
Steps to Reproduce:
I have tested this in my test env on azure. These are the steps which I have followed:
- Podman version should be >= 1.19
- podman login asmereg.azurecr.io -u asmereg -p <passwd>
- podman login registry.redhat.io
- podman run -p50051:50051 -it registry.redhat.io/redhat/redhat-operator-index:v4.7
- opm index prune -f registry.redhat.io/redhat/redhat-operator-index:v4.7 -p advanced-cluster-management,jaeger-product,quay-operator -t asmereg.azurecr.io/redhat/redhat-operator-index:v4.7
- podman push asmereg.azurecr.io/redhat/redhat-operator-index:v4.7
- REG_CREDS=/run/containers/0/auth.json
- oc adm catalog mirror asmereg.azurecr.io/redhat/redhat-operator-index:v4.7 asmereg.azurecr.io -a ${REG_CREDS}
Actual results:
~~~
/title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>Request Header Or Cookie Too Large</center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n"
error: unable to push manifest to asmereg.azurecr.io/rhacm2/thanos-rhel7:1eb45703: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>400 Request Header Or Cookie Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>Request Header Or Cookie Too Large</center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n"
error: unable to push manifest to asmereg.azurecr.io/rhacm2/thanos-rhel7:47118979: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>400 Request Header Or Cookie Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>Request Header Or Cookie Too Large</center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n"
info: Mirroring completed in 15m51.02s (0B/s)
error mirroring image: one or more errors occurred
no digest mapping available for asmereg.azurecr.io/redhat/redhat-operator-index:v4.7, skip writing to ImageContentSourcePolicy
wrote mirroring manifests to manifests-redhat-operator-index-1617354430
~~~
Expected results:
It should mirror images without any errors.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2021:2438
Description of problem: So when we are trying to mirror the image from the pruned image to private registry by running the below command : oc adm catalog mirror \ xyz.azurecr.io/redhat/redhat-operator-index:v4.6 \ xyz.azurecr.io \ -a ${REG_CREDS} \ --filter-by-os='.*' I get the below error for all the images that we are trying to push. below is just 1 sample. error: unable to push manifest to xyz.azurecr.io/rhel8/skopeo:9fc07dfb: Put "https://xyz.azurecr.io/v2/rhel8/skopeo/manifests/9fc07dfb": error parsing HTTP 414 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>414 Request-URI Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>414 Request-URI Too Large</h1></center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n" Version-Release number of selected component (if applicable): - OCP 4.6 How reproducible: - everytime Steps to Reproduce: I have tested this in my test env on azure. These are the steps which I have followed: - Podman version should be >= 1.19 - podman login asmereg.azurecr.io -u asmereg -p <passwd> - podman login registry.redhat.io - podman run -p50051:50051 -it registry.redhat.io/redhat/redhat-operator-index:v4.7 - opm index prune -f registry.redhat.io/redhat/redhat-operator-index:v4.7 -p advanced-cluster-management,jaeger-product,quay-operator -t asmereg.azurecr.io/redhat/redhat-operator-index:v4.7 - podman push asmereg.azurecr.io/redhat/redhat-operator-index:v4.7 - REG_CREDS=/run/containers/0/auth.json - oc adm catalog mirror asmereg.azurecr.io/redhat/redhat-operator-index:v4.7 asmereg.azurecr.io -a ${REG_CREDS} Actual results: ~~~ /title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>Request Header Or Cookie Too Large</center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n" error: unable to push manifest to asmereg.azurecr.io/rhacm2/thanos-rhel7:1eb45703: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>400 Request Header Or Cookie Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>Request Header Or Cookie Too Large</center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n" error: unable to push manifest to asmereg.azurecr.io/rhacm2/thanos-rhel7:47118979: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>400 Request Header Or Cookie Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>Request Header Or Cookie Too Large</center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n" info: Mirroring completed in 15m51.02s (0B/s) error mirroring image: one or more errors occurred no digest mapping available for asmereg.azurecr.io/redhat/redhat-operator-index:v4.7, skip writing to ImageContentSourcePolicy wrote mirroring manifests to manifests-redhat-operator-index-1617354430 ~~~ Expected results: It should mirror images without any errors.