Bug 1399559

Summary:	Can't prune image by command 'oadm prune images' with S3 as the bankend
Product:	OpenShift Container Platform	Reporter:	zhou ying <yinzhou>
Component:	Image Registry	Assignee:	Michal Fojtik <mfojtik>
Status:	CLOSED NOTABUG	QA Contact:	Wei Sun <wsun>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	3.2.1	CC:	aos-bugs, maszulik, mfojtik, tdawson
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	undefined	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-12-19 10:16:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description zhou ying 2016-11-29 09:59:34 UTC

Description of problem:
Use command 'oadm prune image' to prune the image from docker-registry, met error: 500 Internal Server Error

Version-Release number of selected component (if applicable):
openshift version
openshift v3.2.1.17
kubernetes v1.2.0-36-g4a3f9c5
etcd3
docker version: 1.10.3
registry version=v2.1.0

How reproducible:
Always

Steps to Reproduce:
1. Login OpenShift and create project;
2. Create app and do sti build;
3. Delete the app;
4. Use command to delete image:
   `oadm prune images --keep-younger-than=0 --keep-tag-revisions=1  --confirm`


Actual results:
4.  oadm prune images --keep-younger-than=0 --keep-tag-revisions=1  --confirm
I1129 03:30:21.838186    8381 imagepruner.go:430] Unable to find image "sha256:692a1af18dba6314fc8dc3b51ca6930aff50ed3371b9453f4c326922acffcd88" in the graph
I1129 03:30:21.838633    8381 imagepruner.go:430] Unable to find image "sha256:692a1af18dba6314fc8dc3b51ca6930aff50ed3371b9453f4c326922acffcd88" in the graph
I1129 03:30:21.838942    8381 imagepruner.go:430] Unable to find image "sha256:692a1af18dba6314fc8dc3b51ca6930aff50ed3371b9453f4c326922acffcd88" in the graph

Deleting registry layer blobs ...
BLOB
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
error deleting blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 from the registry: 500 Internal Server Error
sha256:b460355310de9d8a1b69455582588e48e6bc3565ef91b5a76c593266d6901921
error deleting blob sha256:b460355310de9d8a1b69455582588e48e6bc3565ef91b5a76c593266d6901921 from the registry: 500 Internal Server Error
sha256:279efb65a614860830f59f0f6eff05c76bfe0cc123ca0a13a946d845c5b7b055
error deleting blob sha256:279efb65a614860830f59f0f6eff05c76bfe0cc123ca0a13a946d845c5b7b055 from the registry: 500 Internal Server Error
sha256:7475eb29149fd39e08d42abb7d4e19228c0be35dd23f1175b295289981a7624c
error deleting blob sha256:7475eb29149fd39e08d42abb7d4e19228c0be35dd23f1175b295289981a7624c from the registry: 500 Internal Server Error
sha256:60f108d9a74364a5d44015ff3f5982cecfdf987eae1f4c92b79931ff49ce7b80
error deleting blob sha256:60f108d9a74364a5d44015ff3f5982cecfdf987eae1f4c92b79931ff49ce7b80 from the registry: 500 Internal Server Error
sha256:b09f30f51562b64ac53704baf81ede4f3f728b3cfd6e36637059ac9e8696b087
error deleting blob sha256:b09f30f51562b64ac53704baf81ede4f3f728b3cfd6e36637059ac9e8696b087 from the registry: 500 Internal Server Error
sha256:81330769d45009a040adc775a1bf9607f30c4b3f8361d40f8f95b628e503d862
error deleting blob sha256:81330769d45009a040adc775a1bf9607f30c4b3f8361d40f8f95b628e503d862 from the registry: 500 Internal Server Error
sha256:8d30e94188e7f13642d975e70c484e48c33867f3ede3277df1145803fa996ac1
error deleting blob sha256:8d30e94188e7f13642d975e70c484e48c33867f3ede3277df1145803fa996ac1 from the registry: 500 Internal Server Error
sha256:9e0d025d6982a31ae6eff14fbff14a879040640dd2220928415454aad8ad3e8d
error deleting blob sha256:9e0d025d6982a31ae6eff14fbff14a879040640dd2220928415454aad8ad3e8d from the registry: 500 Internal Server Error
sha256:e7c36c73b545d32b5a78ee50d87f356f63fd633f2f602f49e8d97329df6107ef
error deleting blob sha256:e7c36c73b545d32b5a78ee50d87f356f63fd633f2f602f49e8d97329df6107ef from the registry: 500 Internal Server Error
sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542
error deleting blob sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542 from the registry: 500 Internal Server Error
error pruning blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4": 500 Internal Server Error
error pruning blob "sha256:b460355310de9d8a1b69455582588e48e6bc3565ef91b5a76c593266d6901921": 500 Internal Server Error
error pruning blob "sha256:279efb65a614860830f59f0f6eff05c76bfe0cc123ca0a13a946d845c5b7b055": 500 Internal Server Error
error pruning blob "sha256:7475eb29149fd39e08d42abb7d4e19228c0be35dd23f1175b295289981a7624c": 500 Internal Server Error
error pruning blob "sha256:60f108d9a74364a5d44015ff3f5982cecfdf987eae1f4c92b79931ff49ce7b80": 500 Internal Server Error
error pruning blob "sha256:b09f30f51562b64ac53704baf81ede4f3f728b3cfd6e36637059ac9e8696b087": 500 Internal Server Error
error pruning blob "sha256:81330769d45009a040adc775a1bf9607f30c4b3f8361d40f8f95b628e503d862": 500 Internal Server Error
error pruning blob "sha256:8d30e94188e7f13642d975e70c484e48c33867f3ede3277df1145803fa996ac1": 500 Internal Server Error
error pruning blob "sha256:9e0d025d6982a31ae6eff14fbff14a879040640dd2220928415454aad8ad3e8d": 500 Internal Server Error
error pruning blob "sha256:e7c36c73b545d32b5a78ee50d87f356f63fd633f2f602f49e8d97329df6107ef": 500 Internal Server Error
error pruning blob "sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542": 500 Internal Server Error


Expected results:
4. Prune the image succeed.

Additional info:
logs from docker-registry's pod:
time="2016-11-29T03:30:21.915924811-05:00" level=debug msg="authorizing request" go.version=go1.4.2 http.request.host="172.31.123.178:5000" http.request.id=0c311006-418b-42fc-9b5a-35ceb4c4aaa0 http.request.method=DELETE http.request.remoteaddr="10.2.1.1:48812" http.request.uri="/admin/blobs/sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" http.request.useragent="Go 1.1 package http" instance.id=e5b82ac1-8c1a-4e84-9d49-7fb5bb4bb8c6 vars.digest="sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" 
time="2016-11-29T03:30:21.916021566-05:00" level=debug msg="Origin auth: checking for access to admin::prune" go.version=go1.4.2 http.request.host="172.31.123.178:5000" http.request.id=0c311006-418b-42fc-9b5a-35ceb4c4aaa0 http.request.method=DELETE http.request.remoteaddr="10.2.1.1:48812" http.request.uri="/admin/blobs/sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" http.request.useragent="Go 1.1 package http" instance.id=e5b82ac1-8c1a-4e84-9d49-7fb5bb4bb8c6 vars.digest="sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" 
time="2016-11-29T03:30:21.92558981-05:00" level=error msg="response completed with error" err.code=UNKNOWN err.detail="error deleting blob \"sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542\": operation unsupported" err.message="unknown error" go.version=go1.4.2 http.request.host="172.31.123.178:5000" http.request.id=0c311006-418b-42fc-9b5a-35ceb4c4aaa0 http.request.method=DELETE http.request.remoteaddr="10.2.1.1:48812" http.request.uri="/admin/blobs/sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" http.request.useragent="Go 1.1 package http" http.response.contenttype="application/json; charset=utf-8" http.response.duration=11.895577ms http.response.status=500 http.response.written=188 instance.id=e5b82ac1-8c1a-4e84-9d49-7fb5bb4bb8c6 vars.digest="sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" 
10.2.1.1 - - [29/Nov/2016:03:30:21 -0500] "DELETE /admin/blobs/sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542 HTTP/1.1" 500 188 "" "Go 1.1 package http"

Comment 1 Maciej Szulik 2016-11-29 10:18:28 UTC

This looks like mis-configuration issue described in https://docs.openshift.org/latest/install_config/registry/registry_known_issues.html#known-issue-prune-fails-due-to-delete-disabled

Comment 2 zhou ying 2016-11-30 05:35:46 UTC

When storage:delete:enabled set to true, re-deploy the registry, and repeat image pruning operation, still met the 500 Internal Server Error:

[root@ip-172-18-13-143 origin]# oc get po 
NAME                      READY     STATUS    RESTARTS   AGE
docker-registry-5-snekh   1/1       Running   0          23m
router-1-6jmx1            1/1       Running   0          2h

[root@ip-172-18-13-143 origin]# oc env po docker-registry-5-snekh --list
# pods docker-registry-5-snekh, container registry
REGISTRY_HTTP_ADDR=:5000
REGISTRY_HTTP_NET=tcp
REGISTRY_HTTP_SECRET=KHPaYvLSbTrvtn2/e2U2EGaKvwlIe6dGKZgtylMN9BQ=
REGISTRY_CONFIGURATION_PATH=/etc/registry/config.yml


[root@ip-172-18-13-143 origin]# oc exec -it docker-registry-5-snekh more /etc/registry/config.yml
version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  delete:
    enabled: true
  redirect:
    disable: false
  cache:
    layerinfo: inmemory
version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    layerinfo: inmemory
  s3:

Comment 3 Maciej Szulik 2016-11-30 11:17:09 UTC

It looks lie your config has double values, you have two version, storage and other stanzas. This might cause the registry picking the 2nd version missing the storage:delete:enabled:true. Please fix the config an re-try.

Comment 4 zhou ying 2016-12-01 02:07:18 UTC

 Maciej Szulik 

   Thank you! After updated the config, the prune has succeed, thanks.


[root@ip-172-18-12-100 origin]# oadm prune  images --keep-younger-than=0 --confirm

Deleting registry layer blobs ...
BLOB
sha256:f432a3a34b9a276fa0f3dc030f3be9b7788f55f6b07bf7cde571fff8dc566832
sha256:3b31813c2cd47f24a2fb54316e88b79863ca305f6e5f33eaecde0b8405180aa6
sha256:8a0dc5f10b5005e3c125afff69192ba424ae71ccd1dd5c938a6f33b189deb003
sha256:08d48e6f1cff259389732d35307bb877215fa28867cdaff50c1dbd6e0b993c1f
sha256:91ec6240a2b5064f5941a7ca7959f597ffd80d2d4a997bad5e45ab8b88ca6984
sha256:b429eff5f252721885900f652c7d32c3cc1185cc8e3426ae4ca7cc6b4ad1a42e
sha256:14f9e16b6593712bc9907588cd47284ea850d0664f4aff5a852a3509386a272b
sha256:a7033a977215eb3b4f7b128c5bb9968870ff0c6f2f3e99922145c5897b00bbbf
sha256:e4e9658e120f74a7a0854025667bbbe23c2102401671df3a64aa0425f17e65b4
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4

Deleting images from server ...
IMAGE
sha256:ff860b45bd20ea91002c39901c02477e25804cbf5ac6cd23f0116780eff2e165


[root@ip-172-18-12-100 origin]# oc exec -it docker-registry-5-s6y1k more /etc/registry/config.yml
version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  delete:
    enabled: true
  redirect:
    disable: false
  cache:
    layerinfo: inmemory
  s3:

Comment 5 Troy Dawson 2016-12-16 21:13:12 UTC

Double checking - can this be closed with notabug?

Comment 6 Maciej Szulik 2016-12-19 10:16:11 UTC

Yup, closing.