Bug 1885179

Summary: oc image append fails on push (uploading a new layer)
Product: OpenShift Container Platform Reporter: Ricardo Maraschini <rmarasch>
Component: ocAssignee: Sally <somalley>
Status: CLOSED ERRATA QA Contact: RamaKasturi <knarra>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.6CC: aos-bugs, jokerman, knarra, mfojtik
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:22:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ricardo Maraschini 2020-10-05 10:58:34 UTC
Description of problem:

oc image append fails when pushing a new layer to an image hosted in quay.io or docker.io registries.


Version-Release number of selected component (if applicable):

Client Version: 4.6.0-fc.8

How reproducible:

Always

Steps to Reproduce:
1. Create a new tar for the new layer
  1.1 mkdir layer
  1.2 cd layer
  1.2 echo test > some-file
  1.3 tar -cvf new-layer.tar some-file
2. Upload the layer:
  2.1 oc image append --from=quay.io/rmarasch/my-image:latest --to=quay.io/rmarasch/my-image:append new-layer.tar
  2.2 repository above exists

Actual results:

Upload to quay.io fails with

Uploading ... error: unable to upload new layer (0): Patch "https://quay.io/v2/rmarasch/my-image/blobs/uploads/9ecb7583-2060-4f1d-8ccc-71a5141cd462": stream error: stream ID 17; PROTOCOL_ERROR


Upload to docker.io fails with

Uploading ... error: unable to upload new layer (0): received unexpected HTTP status: 504 Gateway Time-out


Expected results:

New layer is uploaded successfully. 

Additional info:

Comment 1 Ricardo Maraschini 2020-10-06 08:45:29 UTC
It turns out that the image layer I was trying to upload was considered invalid (not a gzip'd one). The error message was misguiding, I have proposed a fix for this on https://github.com/openshift/oc/pull/604

Comment 3 RamaKasturi 2020-10-28 07:13:40 UTC
Verified with the payload below and i see that error message is clear now when uploading a layer which is not a gzip'd one.

[ramakasturinarra@dhcp35-60 openshift-client-linux-4.7.0-0.nightly-2020-10-27-051128]$ ./oc version
Client Version: 4.7.0-0.nightly-2020-10-27-051128
Server Version: 4.7.0-0.nightly-2020-10-27-051128
Kubernetes Version: v1.19.0+e67f5dc


when uploading a layer which is not a gziped one below error is seen with latest oc client

[ramakasturinarra@dhcp35-60 openshift-client-linux-4.7.0-0.nightly-2020-10-27-051128]$ sudo ./oc image append --from=quay.io/knarra/my-kube-scheduler43:latest --to=quay.io/knarra/my-kube-scheduler43:append layer/new-layer.tar
Uploading ... error: unable to upload new layer (0): Patch "https://quay.io/v2/knarra/my-kube-scheduler43/blobs/uploads/02bebf5f-9e1c-4e15-97f4-1316fe04b8bb": unable to create gzip reader layer upload: gzip: invalid header

when uploading a layer which is gziped, appened works fine with out any issues
sudo ./oc image append --from=quay.io/knarra/testcli:test --to=quay.io/knarra/testcli:append newdir.tar.gz
Uploading ... 69.42B/s
Pushed sha256:af2ae401755f2db5bd8db94f8a2066ceebf9ab751aa425072fba80ad012e2989 to quay.io/knarra/testcli:append


When uploading a layer which is not a gziped one error was misguiding in older oc client
[ramakasturinarra@dhcp35-60 openshift-client-linux-4.7.0-0.nightly-2020-10-27-051128]$ oc version
Client Version: 4.6.0-202009281501.p0-61364f0

[ramakasturinarra@dhcp35-60 openshift-client-linux-4.7.0-0.nightly-2020-10-27-051128]$ sudo oc image append --from=quay.io/knarra/my-kube-scheduler43:latest --to=quay.io/knarra/my-kube-scheduler43:append layer/new-layer.tar
Uploading ... error: unable to upload new layer (0): Patch "https://quay.io/v2/knarra/my-kube-scheduler43/blobs/uploads/13eb5d10-74de-43a6-9117-3f88b25e27e7": stream error: stream ID 17; PROTOCOL_ERROR

Based on the above moving bug to verified state.

Comment 6 errata-xmlrpc 2021-02-24 15:22:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633