Bug 1668413 - [4.0] When using the oc new-app to create a new build, the builds are creating incomplete multipart uploads to S3 and incomplete uploads are not cleaned automatically
Summary: [4.0] When using the oc new-app to create a new build, the builds are creatin...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.1.0
Assignee: Oleg Bulatov
QA Contact: wewang
URL:
Whiteboard:
Depends On: 1662132
Blocks: 1668411 1668412
TreeView+ depends on / blocked
 
Reported: 2019-01-22 16:53 UTC by Oleg Bulatov
Modified: 2019-06-04 10:42 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the cluster role system:image-pruner was required for all DELETE requests to the registry. Consequence: the regular client couldn't cancel its uploads and S3 multipart uploads was piling up. Fix: accept DELETE requests for uploads from clients who are allowed to write into them. Result: clients are able to cancel their uploads.
Clone Of: 1662132
Environment:
Last Closed: 2019-06-04 10:42:08 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:42:23 UTC

Comment 3 wewang 2019-01-25 10:18:49 UTC
Tested in 
openshift-install v0.10.1
payload: registry.svc.ci.openshift.org/ocp/release@sha256:9185e93b4cf65abe8712b2e489226406c3ea9406da8051c8ae201a9159fa3db8

still got  below blobs/uploads info, it should not have these info in registry log right? 

$ oc logs -f pod/image-registry-5bcb79bf8d-t4wcr -n openshift-image-registry  |grep -i upload
time="2019-01-25T09:50:16.085978913Z" level=info msg="authorized request" go.version=go1.9.4 http.request.contenttype=application/octet-stream http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=1419d891-6216-41ac-b665-457c5c4e7842 http.request.method=PUT http.request.remoteaddr="10.131.2.9:33184" http.request.uri="/v2/wewang1/wangtest/blobs/uploads/5bd10630-7524-426b-8330-82f3e277b79d?_state=2VtkimlFSqBQSh7xJSrxwe0rjMIiu4JIZ0C7KdWZ3Kd7Ik5hbWUiOiJ3ZXdhbmcxL3dhbmd0ZXN0IiwiVVVJRCI6IjViZDEwNjMwLTc1MjQtNDI2Yi04MzMwLTgyZjNlMjc3Yjc5ZCIsIk9mZnNldCI6MTczNjgsIlN0YXJ0ZWRBdCI6IjIwMTktMDEtMjVUMDk6NTA6MTVaIn0%3D&digest=sha256%3A9f5b143865a76f8cdaf11f5b0358bb4f942b9f7be15ec686549997fcf86193da" http.request.useragent=Go-http-client/1.1 openshift.auth.user="system:serviceaccount:wewang1:builder" vars.name=wewang1/wangtest vars.uuid=5bd10630-7524-426b-8330-82f3e277b79d
time="2019-01-25T09:50:16.395426311Z" level=info msg="response completed" go.version=go1.9.4 http.request.contenttype=application/octet-stream http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=1419d891-6216-41ac-b665-457c5c4e7842 http.request.method=PUT http.request.remoteaddr="10.131.2.9:33184" http.request.uri="/v2/wewang1/wangtest/blobs/uploads/5bd10630-7524-426b-8330-82f3e277b79d?_state=2VtkimlFSqBQSh7xJSrxwe0rjMIiu4JIZ0C7KdWZ3Kd7Ik5hbWUiOiJ3ZXdhbmcxL3dhbmd0ZXN0IiwiVVVJRCI6IjViZDEwNjMwLTc1MjQtNDI2Yi04MzMwLTgyZjNlMjc3Yjc5ZCIsIk9mZnNldCI6MTczNjgsIlN0YXJ0ZWRBdCI6IjIwMTktMDEtMjVUMDk6NTA6MTVaIn0%3D&digest=sha256%3A9f5b143865a76f8cdaf11f5b0358bb4f942b9f7be15ec686549997fcf86193da" http.request.useragent=Go-http-client/1.1 http.response.duration=316.614507ms http.response.status=201 http.response.written=0
time="2019-01-25T09:50:16.395757314Z" level=info msg=response go.version=go1.9.4 http.request.contenttype=application/octet-stream http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=ebbbc68c-4af1-46dd-845d-43dd0f31a277 http.request.method=PUT http.request.remoteaddr="10.131.2.9:33184" http.request.uri="/v2/wewang1/wangtest/blobs/uploads/5bd10630-7524-426b-8330-82f3e277b79d?_state=2VtkimlFSqBQSh7xJSrxwe0rjMIiu4JIZ0C7KdWZ3Kd7Ik5hbWUiOiJ3ZXdhbmcxL3dhbmd0ZXN0IiwiVVVJRCI6IjViZDEwNjMwLTc1MjQtNDI2Yi04MzMwLTgyZjNlMjc3Yjc5ZCIsIk9mZnNldCI6MTczNjgsIlN0YXJ0ZWRBdCI6IjIwMTktMDEtMjVUMDk6NTA6MTVaIn0%3D&digest=sha256%3A9f5b143865a76f8cdaf11f5b0358bb4f942b9f7be15ec686549997fcf86193da" http.request.useragent=Go-http-client/1.1 http.response.duration=316.987566ms http.response.status=201 http.response.written=0

Comment 4 Oleg Bulatov 2019-01-25 13:03:11 UTC
They should be there. These lines mean the client has started uploading a blob.

Comment 5 wewang 2019-01-30 05:51:26 UTC
verified in 
 openshift-install v0.11.0
 payload: quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d
 version: 4.0.0-0.2

steps:
1. Create apps
$oc new-app --name e2e https://github.com/appuio/endtoend-docker-helloworld.git

2. Check pod in openshift-image-registry
$ oc get pods -n openshift-image-registry
NAME                                               READY     STATUS    RESTARTS   AGE
cluster-image-registry-operator-684499b66b-q7p94   1/1       Running   0          20h
image-registry-7748c564d5-z8zk6                    1/1       Running   0          20h

$ oc rsh image-registry-7748c564d5-z8zk6
sh-4.2$ ls
bin  boot  config.yml  dev  etc  home  lib  lib64  media  mnt  opt  proc  registry  root  run  sbin  srv  sys  tmp  usr  var
sh-4.2$ more config.yml 
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /registry                     
  delete:
    enabled: true

sh-4.2$ cd /registry/                      #there's no uploads in registry directory 
sh-4.2$ ls -al
total 0
drwxr-xr-x. 2 root root  6 Jan 25 19:20 .
drwxr-xr-x. 1 root root 17 Jan 29 09:06 ..

@Oleg Bulatov, do you think it's enough to verify the bug? if not feel free to notice me, thx.

Comment 8 errata-xmlrpc 2019-06-04 10:42:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.