Bug 1668413
| Summary: | [4.0] When using the oc new-app to create a new build, the builds are creating incomplete multipart uploads to S3 and incomplete uploads are not cleaned automatically | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Oleg Bulatov <obulatov> |
| Component: | Build | Assignee: | Oleg Bulatov <obulatov> |
| Status: | CLOSED ERRATA | QA Contact: | wewang <wewang> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.1.0 | CC: | aos-bugs, bparees, wzheng |
| Target Milestone: | --- | ||
| Target Release: | 4.1.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: the cluster role system:image-pruner was required for all DELETE requests to the registry.
Consequence: the regular client couldn't cancel its uploads and S3 multipart uploads was piling up.
Fix: accept DELETE requests for uploads from clients who are allowed to write into them.
Result: clients are able to cancel their uploads.
|
Story Points: | --- |
| Clone Of: | 1662132 | Environment: | |
| Last Closed: | 2019-06-04 10:42:08 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1662132 | ||
| Bug Blocks: | 1668411, 1668412 | ||
|
Comment 1
Oleg Bulatov
2019-01-23 13:18:08 UTC
Tested in openshift-install v0.10.1 payload: registry.svc.ci.openshift.org/ocp/release@sha256:9185e93b4cf65abe8712b2e489226406c3ea9406da8051c8ae201a9159fa3db8 still got below blobs/uploads info, it should not have these info in registry log right? $ oc logs -f pod/image-registry-5bcb79bf8d-t4wcr -n openshift-image-registry |grep -i upload time="2019-01-25T09:50:16.085978913Z" level=info msg="authorized request" go.version=go1.9.4 http.request.contenttype=application/octet-stream http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=1419d891-6216-41ac-b665-457c5c4e7842 http.request.method=PUT http.request.remoteaddr="10.131.2.9:33184" http.request.uri="/v2/wewang1/wangtest/blobs/uploads/5bd10630-7524-426b-8330-82f3e277b79d?_state=2VtkimlFSqBQSh7xJSrxwe0rjMIiu4JIZ0C7KdWZ3Kd7Ik5hbWUiOiJ3ZXdhbmcxL3dhbmd0ZXN0IiwiVVVJRCI6IjViZDEwNjMwLTc1MjQtNDI2Yi04MzMwLTgyZjNlMjc3Yjc5ZCIsIk9mZnNldCI6MTczNjgsIlN0YXJ0ZWRBdCI6IjIwMTktMDEtMjVUMDk6NTA6MTVaIn0%3D&digest=sha256%3A9f5b143865a76f8cdaf11f5b0358bb4f942b9f7be15ec686549997fcf86193da" http.request.useragent=Go-http-client/1.1 openshift.auth.user="system:serviceaccount:wewang1:builder" vars.name=wewang1/wangtest vars.uuid=5bd10630-7524-426b-8330-82f3e277b79d time="2019-01-25T09:50:16.395426311Z" level=info msg="response completed" go.version=go1.9.4 http.request.contenttype=application/octet-stream http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=1419d891-6216-41ac-b665-457c5c4e7842 http.request.method=PUT http.request.remoteaddr="10.131.2.9:33184" http.request.uri="/v2/wewang1/wangtest/blobs/uploads/5bd10630-7524-426b-8330-82f3e277b79d?_state=2VtkimlFSqBQSh7xJSrxwe0rjMIiu4JIZ0C7KdWZ3Kd7Ik5hbWUiOiJ3ZXdhbmcxL3dhbmd0ZXN0IiwiVVVJRCI6IjViZDEwNjMwLTc1MjQtNDI2Yi04MzMwLTgyZjNlMjc3Yjc5ZCIsIk9mZnNldCI6MTczNjgsIlN0YXJ0ZWRBdCI6IjIwMTktMDEtMjVUMDk6NTA6MTVaIn0%3D&digest=sha256%3A9f5b143865a76f8cdaf11f5b0358bb4f942b9f7be15ec686549997fcf86193da" http.request.useragent=Go-http-client/1.1 http.response.duration=316.614507ms http.response.status=201 http.response.written=0 time="2019-01-25T09:50:16.395757314Z" level=info msg=response go.version=go1.9.4 http.request.contenttype=application/octet-stream http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=ebbbc68c-4af1-46dd-845d-43dd0f31a277 http.request.method=PUT http.request.remoteaddr="10.131.2.9:33184" http.request.uri="/v2/wewang1/wangtest/blobs/uploads/5bd10630-7524-426b-8330-82f3e277b79d?_state=2VtkimlFSqBQSh7xJSrxwe0rjMIiu4JIZ0C7KdWZ3Kd7Ik5hbWUiOiJ3ZXdhbmcxL3dhbmd0ZXN0IiwiVVVJRCI6IjViZDEwNjMwLTc1MjQtNDI2Yi04MzMwLTgyZjNlMjc3Yjc5ZCIsIk9mZnNldCI6MTczNjgsIlN0YXJ0ZWRBdCI6IjIwMTktMDEtMjVUMDk6NTA6MTVaIn0%3D&digest=sha256%3A9f5b143865a76f8cdaf11f5b0358bb4f942b9f7be15ec686549997fcf86193da" http.request.useragent=Go-http-client/1.1 http.response.duration=316.987566ms http.response.status=201 http.response.written=0 They should be there. These lines mean the client has started uploading a blob. verified in openshift-install v0.11.0 payload: quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d version: 4.0.0-0.2 steps: 1. Create apps $oc new-app --name e2e https://github.com/appuio/endtoend-docker-helloworld.git 2. Check pod in openshift-image-registry $ oc get pods -n openshift-image-registry NAME READY STATUS RESTARTS AGE cluster-image-registry-operator-684499b66b-q7p94 1/1 Running 0 20h image-registry-7748c564d5-z8zk6 1/1 Running 0 20h $ oc rsh image-registry-7748c564d5-z8zk6 sh-4.2$ ls bin boot config.yml dev etc home lib lib64 media mnt opt proc registry root run sbin srv sys tmp usr var sh-4.2$ more config.yml storage: cache: blobdescriptor: inmemory filesystem: rootdirectory: /registry delete: enabled: true sh-4.2$ cd /registry/ #there's no uploads in registry directory sh-4.2$ ls -al total 0 drwxr-xr-x. 2 root root 6 Jan 25 19:20 . drwxr-xr-x. 1 root root 17 Jan 29 09:06 .. @Oleg Bulatov, do you think it's enough to verify the bug? if not feel free to notice me, thx. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758 |