Hide Forgot
Description of problem: Use command 'oadm prune image' to prune the image from docker-registry, met error: 500 Internal Server Error Version-Release number of selected component (if applicable): openshift version openshift v3.2.1.17 kubernetes v1.2.0-36-g4a3f9c5 etcd3 docker version: 1.10.3 registry version=v2.1.0 How reproducible: Always Steps to Reproduce: 1. Login OpenShift and create project; 2. Create app and do sti build; 3. Delete the app; 4. Use command to delete image: `oadm prune images --keep-younger-than=0 --keep-tag-revisions=1 --confirm` Actual results: 4. oadm prune images --keep-younger-than=0 --keep-tag-revisions=1 --confirm I1129 03:30:21.838186 8381 imagepruner.go:430] Unable to find image "sha256:692a1af18dba6314fc8dc3b51ca6930aff50ed3371b9453f4c326922acffcd88" in the graph I1129 03:30:21.838633 8381 imagepruner.go:430] Unable to find image "sha256:692a1af18dba6314fc8dc3b51ca6930aff50ed3371b9453f4c326922acffcd88" in the graph I1129 03:30:21.838942 8381 imagepruner.go:430] Unable to find image "sha256:692a1af18dba6314fc8dc3b51ca6930aff50ed3371b9453f4c326922acffcd88" in the graph Deleting registry layer blobs ... BLOB sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 error deleting blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 from the registry: 500 Internal Server Error sha256:b460355310de9d8a1b69455582588e48e6bc3565ef91b5a76c593266d6901921 error deleting blob sha256:b460355310de9d8a1b69455582588e48e6bc3565ef91b5a76c593266d6901921 from the registry: 500 Internal Server Error sha256:279efb65a614860830f59f0f6eff05c76bfe0cc123ca0a13a946d845c5b7b055 error deleting blob sha256:279efb65a614860830f59f0f6eff05c76bfe0cc123ca0a13a946d845c5b7b055 from the registry: 500 Internal Server Error sha256:7475eb29149fd39e08d42abb7d4e19228c0be35dd23f1175b295289981a7624c error deleting blob sha256:7475eb29149fd39e08d42abb7d4e19228c0be35dd23f1175b295289981a7624c from the registry: 500 Internal Server Error sha256:60f108d9a74364a5d44015ff3f5982cecfdf987eae1f4c92b79931ff49ce7b80 error deleting blob sha256:60f108d9a74364a5d44015ff3f5982cecfdf987eae1f4c92b79931ff49ce7b80 from the registry: 500 Internal Server Error sha256:b09f30f51562b64ac53704baf81ede4f3f728b3cfd6e36637059ac9e8696b087 error deleting blob sha256:b09f30f51562b64ac53704baf81ede4f3f728b3cfd6e36637059ac9e8696b087 from the registry: 500 Internal Server Error sha256:81330769d45009a040adc775a1bf9607f30c4b3f8361d40f8f95b628e503d862 error deleting blob sha256:81330769d45009a040adc775a1bf9607f30c4b3f8361d40f8f95b628e503d862 from the registry: 500 Internal Server Error sha256:8d30e94188e7f13642d975e70c484e48c33867f3ede3277df1145803fa996ac1 error deleting blob sha256:8d30e94188e7f13642d975e70c484e48c33867f3ede3277df1145803fa996ac1 from the registry: 500 Internal Server Error sha256:9e0d025d6982a31ae6eff14fbff14a879040640dd2220928415454aad8ad3e8d error deleting blob sha256:9e0d025d6982a31ae6eff14fbff14a879040640dd2220928415454aad8ad3e8d from the registry: 500 Internal Server Error sha256:e7c36c73b545d32b5a78ee50d87f356f63fd633f2f602f49e8d97329df6107ef error deleting blob sha256:e7c36c73b545d32b5a78ee50d87f356f63fd633f2f602f49e8d97329df6107ef from the registry: 500 Internal Server Error sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542 error deleting blob sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542 from the registry: 500 Internal Server Error error pruning blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4": 500 Internal Server Error error pruning blob "sha256:b460355310de9d8a1b69455582588e48e6bc3565ef91b5a76c593266d6901921": 500 Internal Server Error error pruning blob "sha256:279efb65a614860830f59f0f6eff05c76bfe0cc123ca0a13a946d845c5b7b055": 500 Internal Server Error error pruning blob "sha256:7475eb29149fd39e08d42abb7d4e19228c0be35dd23f1175b295289981a7624c": 500 Internal Server Error error pruning blob "sha256:60f108d9a74364a5d44015ff3f5982cecfdf987eae1f4c92b79931ff49ce7b80": 500 Internal Server Error error pruning blob "sha256:b09f30f51562b64ac53704baf81ede4f3f728b3cfd6e36637059ac9e8696b087": 500 Internal Server Error error pruning blob "sha256:81330769d45009a040adc775a1bf9607f30c4b3f8361d40f8f95b628e503d862": 500 Internal Server Error error pruning blob "sha256:8d30e94188e7f13642d975e70c484e48c33867f3ede3277df1145803fa996ac1": 500 Internal Server Error error pruning blob "sha256:9e0d025d6982a31ae6eff14fbff14a879040640dd2220928415454aad8ad3e8d": 500 Internal Server Error error pruning blob "sha256:e7c36c73b545d32b5a78ee50d87f356f63fd633f2f602f49e8d97329df6107ef": 500 Internal Server Error error pruning blob "sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542": 500 Internal Server Error Expected results: 4. Prune the image succeed. Additional info: logs from docker-registry's pod: time="2016-11-29T03:30:21.915924811-05:00" level=debug msg="authorizing request" go.version=go1.4.2 http.request.host="172.31.123.178:5000" http.request.id=0c311006-418b-42fc-9b5a-35ceb4c4aaa0 http.request.method=DELETE http.request.remoteaddr="10.2.1.1:48812" http.request.uri="/admin/blobs/sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" http.request.useragent="Go 1.1 package http" instance.id=e5b82ac1-8c1a-4e84-9d49-7fb5bb4bb8c6 vars.digest="sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" time="2016-11-29T03:30:21.916021566-05:00" level=debug msg="Origin auth: checking for access to admin::prune" go.version=go1.4.2 http.request.host="172.31.123.178:5000" http.request.id=0c311006-418b-42fc-9b5a-35ceb4c4aaa0 http.request.method=DELETE http.request.remoteaddr="10.2.1.1:48812" http.request.uri="/admin/blobs/sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" http.request.useragent="Go 1.1 package http" instance.id=e5b82ac1-8c1a-4e84-9d49-7fb5bb4bb8c6 vars.digest="sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" time="2016-11-29T03:30:21.92558981-05:00" level=error msg="response completed with error" err.code=UNKNOWN err.detail="error deleting blob \"sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542\": operation unsupported" err.message="unknown error" go.version=go1.4.2 http.request.host="172.31.123.178:5000" http.request.id=0c311006-418b-42fc-9b5a-35ceb4c4aaa0 http.request.method=DELETE http.request.remoteaddr="10.2.1.1:48812" http.request.uri="/admin/blobs/sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" http.request.useragent="Go 1.1 package http" http.response.contenttype="application/json; charset=utf-8" http.response.duration=11.895577ms http.response.status=500 http.response.written=188 instance.id=e5b82ac1-8c1a-4e84-9d49-7fb5bb4bb8c6 vars.digest="sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542" 10.2.1.1 - - [29/Nov/2016:03:30:21 -0500] "DELETE /admin/blobs/sha256:c00f9a1266846ed8c81fbe618f1b7e5fe4954b3a3c8fa494422b52e8ef034542 HTTP/1.1" 500 188 "" "Go 1.1 package http"
This looks like mis-configuration issue described in https://docs.openshift.org/latest/install_config/registry/registry_known_issues.html#known-issue-prune-fails-due-to-delete-disabled
When storage:delete:enabled set to true, re-deploy the registry, and repeat image pruning operation, still met the 500 Internal Server Error: [root@ip-172-18-13-143 origin]# oc get po NAME READY STATUS RESTARTS AGE docker-registry-5-snekh 1/1 Running 0 23m router-1-6jmx1 1/1 Running 0 2h [root@ip-172-18-13-143 origin]# oc env po docker-registry-5-snekh --list # pods docker-registry-5-snekh, container registry REGISTRY_HTTP_ADDR=:5000 REGISTRY_HTTP_NET=tcp REGISTRY_HTTP_SECRET=KHPaYvLSbTrvtn2/e2U2EGaKvwlIe6dGKZgtylMN9BQ= REGISTRY_CONFIGURATION_PATH=/etc/registry/config.yml [root@ip-172-18-13-143 origin]# oc exec -it docker-registry-5-snekh more /etc/registry/config.yml version: 0.1 log: level: debug http: addr: :5000 storage: delete: enabled: true redirect: disable: false cache: layerinfo: inmemory version: 0.1 log: level: debug http: addr: :5000 storage: cache: layerinfo: inmemory s3:
It looks lie your config has double values, you have two version, storage and other stanzas. This might cause the registry picking the 2nd version missing the storage:delete:enabled:true. Please fix the config an re-try.
Maciej Szulik Thank you! After updated the config, the prune has succeed, thanks. [root@ip-172-18-12-100 origin]# oadm prune images --keep-younger-than=0 --confirm Deleting registry layer blobs ... BLOB sha256:f432a3a34b9a276fa0f3dc030f3be9b7788f55f6b07bf7cde571fff8dc566832 sha256:3b31813c2cd47f24a2fb54316e88b79863ca305f6e5f33eaecde0b8405180aa6 sha256:8a0dc5f10b5005e3c125afff69192ba424ae71ccd1dd5c938a6f33b189deb003 sha256:08d48e6f1cff259389732d35307bb877215fa28867cdaff50c1dbd6e0b993c1f sha256:91ec6240a2b5064f5941a7ca7959f597ffd80d2d4a997bad5e45ab8b88ca6984 sha256:b429eff5f252721885900f652c7d32c3cc1185cc8e3426ae4ca7cc6b4ad1a42e sha256:14f9e16b6593712bc9907588cd47284ea850d0664f4aff5a852a3509386a272b sha256:a7033a977215eb3b4f7b128c5bb9968870ff0c6f2f3e99922145c5897b00bbbf sha256:e4e9658e120f74a7a0854025667bbbe23c2102401671df3a64aa0425f17e65b4 sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 Deleting images from server ... IMAGE sha256:ff860b45bd20ea91002c39901c02477e25804cbf5ac6cd23f0116780eff2e165 [root@ip-172-18-12-100 origin]# oc exec -it docker-registry-5-s6y1k more /etc/registry/config.yml version: 0.1 log: level: debug http: addr: :5000 storage: delete: enabled: true redirect: disable: false cache: layerinfo: inmemory s3:
Double checking - can this be closed with notabug?
Yup, closing.