Bug 1754042
| Summary: | [gcp] `oc adm release new` fails if upload to GCS storage fails | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Adam Kaplan <adam.kaplan> | ||||
| Component: | Image Registry | Assignee: | Clayton Coleman <ccoleman> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Wenjing Zheng <wzheng> | ||||
| Severity: | low | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.3.0 | CC: | aos-bugs, ccoleman, hwmwm, obulatov, wking | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.3.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-01-23 11:06:39 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Adam Kaplan
2019-09-20 15:59:45 UTC
CC Trevor King - it may be more realistic for `oc adm release new` to be updated to retry in situations like this. There's not much the registry can do if the backend storage drops out. We need to update all the image commands to retry a few times. There's some in mirror, some here, but in general this is a broad sweeping problem with dealing with problematic registries. One problem is that if you get a 500 uploading a 1gb file, do you *really* want to retry that a second time? Absolutely, just resume your push [1]. [1]: https://github.com/opencontainers/distribution-spec/blob/v1.0.0-rc0/spec.md#initiate-resumable-blob-upload Clients usually send a layer as a single chunk, so uploads can't be really resumed (technically they can be resumed, but only from the first byte). > Clients usually send a layer as a single chunk... Which clients? Looks like containers/image doesn't support this yet [1], but I don't see why it couldn't learn about it. I don't have the Quay or OpenShift registry code handy to grep through for server-side support, but Blob-Upload-UUID is clearly part of the distribution spec. [1]: https://github.com/containers/image/search?utf8=%E2%9C%93&q=Blob-Upload-UUID&type= All clients that I know. https://github.com/docker/distribution/blob/master/registry/client/blob_writer.go (this is used by Docker, ReadFrom sends the entire blob and it doesn't support the Content-Range header) https://github.com/containers/image/blob/a911b201c9edde74bfaabc85a61e85cb60466dff/docker/docker_image_dest.go#L141 (chunked upload is not supported either, also io.Reader is not seekable, so if some last bytes are lost on the server side, they can't be reread and resend) On the server side Distribution has problems as well due to the eventually consistent nature of S3. Yeah without registry/client support I'm hesitant to go down that path. Note that we *should*, just not really a good use of time. I think we could easily do a time based retry (track how long previous took, then retry if below some bounded attempt. The point of retries is to reduce overall prob of failure. Hi Oleg, I can get 500 internal error but not sure whether oc re-tries: https://privatebin-it-iso.int.open.paas.redhat.com/?dacaec7d6f1952e1#7a9UQOx7a+RxQzTZtFmKirEfduiWwWTBxS1NRVN4RKs= Whether just try with 500 error is enough? it seems hard to get other error code. Created attachment 1633599 [details]
Request twice log
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062 oc PR landed in 99a4f81 [1]. Checking to see when this went out: $ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.3.0-rc.0 | grep cli-artifacts cli-artifacts https://github.com/openshift/oc 6a937dfe56ff26255d09702c69b8406040c14505 $ git log --oneline origin/release-4.3 | grep -n '6a937df\|99a4f81' 5:6a937dfe5 Merge pull request #208 from openshift-cherrypick-robot/cherry-pick-207-to-release-4.3 124:99a4f8118 Merge pull request #112 from smarterclayton/retry_error So yeah, this was well before 4.3.0-rc.0. And as yet no backport to 4.2. [1]: https://github.com/openshift/oc/pull/112#event-2686554337 |