Bug 1946839 - [Azure] While mirroring images to private registry throwing error: invalid character '<' looking for beginning of value
Summary: [Azure] While mirroring images to private registry throwing error: invalid ch...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: Maciej Szulik
QA Contact: zhou ying
URL:
Whiteboard:
Depends On:
Blocks: 1976284
TreeView+ depends on / blocked
 
Reported: 2021-04-07 01:09 UTC by Neeraj
Modified: 2024-10-01 17:51 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: During mirroring we create an authorization header with all the image information. Consequence: Some registries have limits on header size which are violated by the size. Fix: Force --skip-multiple-scopes for oc adm catalog mirror command. Result: Mirroring works as expected.
Clone Of:
: 1976284 (view as bug list)
Environment:
Last Closed: 2021-07-27 22:57:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 780 0 None open Bug 1946839: Revert: Split work of oc image mirror 2021-04-07 11:57:45 UTC
Red Hat Knowledge Base (Solution) 6135231 0 None None None 2021-06-23 11:20:29 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:58:17 UTC

Description Neeraj 2021-04-07 01:09:22 UTC
Description of problem:


So when we are trying to mirror the image from the pruned image to private registry by running the below command : 
 oc adm catalog mirror \
    xyz.azurecr.io/redhat/redhat-operator-index:v4.6 \
    xyz.azurecr.io \
    -a ${REG_CREDS} \
    --filter-by-os='.*'

I get the below error for all the images that we are trying to push. below is just 1 sample.
error: unable to push manifest to xyz.azurecr.io/rhel8/skopeo:9fc07dfb: Put "https://xyz.azurecr.io/v2/rhel8/skopeo/manifests/9fc07dfb": error parsing HTTP 414 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>414 Request-URI Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>414 Request-URI Too Large</h1></center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n"



Version-Release number of selected component (if applicable):

- OCP 4.6

How reproducible:

- everytime


Steps to Reproduce:

I have tested this in my test env on azure. These are the steps which I have followed:


- Podman version should be >= 1.19

- podman login asmereg.azurecr.io -u asmereg -p <passwd>

- podman login registry.redhat.io

- podman run -p50051:50051 -it registry.redhat.io/redhat/redhat-operator-index:v4.7

- opm index prune  -f registry.redhat.io/redhat/redhat-operator-index:v4.7 -p advanced-cluster-management,jaeger-product,quay-operator -t asmereg.azurecr.io/redhat/redhat-operator-index:v4.7


- podman push  asmereg.azurecr.io/redhat/redhat-operator-index:v4.7

- REG_CREDS=/run/containers/0/auth.json

- oc adm catalog mirror asmereg.azurecr.io/redhat/redhat-operator-index:v4.7 asmereg.azurecr.io -a ${REG_CREDS}



Actual results:
~~~
/title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>Request Header Or Cookie Too Large</center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n"
error: unable to push manifest to asmereg.azurecr.io/rhacm2/thanos-rhel7:1eb45703: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>400 Request Header Or Cookie Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>Request Header Or Cookie Too Large</center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n"
error: unable to push manifest to asmereg.azurecr.io/rhacm2/thanos-rhel7:47118979: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>400 Request Header Or Cookie Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>Request Header Or Cookie Too Large</center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n"
info: Mirroring completed in 15m51.02s (0B/s)
error mirroring image: one or more errors occurred
no digest mapping available for asmereg.azurecr.io/redhat/redhat-operator-index:v4.7, skip writing to ImageContentSourcePolicy
wrote mirroring manifests to manifests-redhat-operator-index-1617354430
~~~


Expected results:

It should mirror images without any errors.

Comment 26 errata-xmlrpc 2021-07-27 22:57:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.