Bug 1237271 - Image layers won't be pruned if the image stream is deleted
Summary: Image layers won't be pruned if the image stream is deleted
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.0.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Andy Goldstein
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On: 1235148
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-30 16:53 UTC by Johnny Liu
Modified: 2017-03-08 18:12 UTC (History)
9 users (show)

Fixed In Version: openshift-3.0.1.0-0.git.133.b1c1f03.el7ose.x86_64.rpm
Doc Type: Bug Fix
Doc Text:
Previously, pruning images associated with an image stream that had been removed would fail. This bug fix updates layer pruning to always delete blobs from the registry, even if the image stream(s) that referenced the layer no longer exists. In the event that there are no longer any image streams referencing the layer, the blob can still be deleted, but not the registry image repository layer link files.
Clone Of: 1235148
Environment:
Last Closed: 2015-08-03 20:17:32 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:1540 normal SHIPPED_LIVE Red Hat OpenShift Enterprise 3.0.1.0 bug fix and enhancement update 2015-08-04 00:17:18 UTC

Comment 1 Andy Goldstein 2015-06-30 17:17:47 UTC
https://github.com/openshift/origin/pull/3515

Comment 3 openshift-github-bot 2015-07-07 15:42:41 UTC
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/963639753dcd53417b7b0293ae9b942c060c65f5
Image pruning improvements

Update layer pruning to always delete blobs from the registry, even if
the image stream(s) that referenced the layer no longer exists. In the
event that there aren't any image streams referencing the layer any
more, we'll be able to delete the blob, but not the registry image
repository layer link files.

Fixes bug 1237271

Extract NewImagePruner arguments to ImagePrunerOptions struct.

Add attempt to check for connectivity to the registry before performing
actual pruning. If the connection attempt fails, report the error and
stop.

Reorganize image pruning operation order:

1. Remove image references from image streams
2. Remove registry data (repo layer links, blobs, repo manifest data)

If any errors occurred in 1 or 2, stop. Otherwise:

3. Remove images from OpenShift

This allows us to preserve images in the event there are issues deleting
from the registry. This way, additional calls to 'oadm prune' can be
attempted and we'll still have the image data to build the graph.

Comment 4 Scott Dodson 2015-07-08 19:28:49 UTC
The referenced pull request is in openshift-3.0.1.0-0.git.133.b1c1f03.el7ose.x86_64.rpm

Puddle: http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/3.0/2015-07-07.1/RH7-RHOSE-3.0/x86_64/os/

Comment 7 Johnny Liu 2015-07-10 11:59:00 UTC
Verified this bug with, and PASS.


On master:
# oadm policy add-cluster-role-to-user system:image-pruner jialiu
# oc expose service docker-registry --hostname=registry.jialiu.com


On client:
$  oc login
$ oc new-project jialiu
$ oc new-app https://github.com/openshift/simple-openshift-sinatra-sti.git  -i openshift/ruby
$ oc delete all --all
$ oadm prune images --keep-younger-than=0 --keep-tag-revisions=1 
Dry run enabled - no modifications will be made. Add --confirm to remove images

Deleting registry layer blobs ...
BLOB
sha256:3f45ac862e1040ba0a9dfd28b5e06c20486b752e85016d12f87f0fb018b4f193
sha256:5880a3208e17b1185f9c933b86b5f7fbd8093c5619f9439d18213ae31a93dcbc
sha256:e5b1e91fc7747f5121550b1f4d605788fdaddef9d6cab3230c79362eab057ada
sha256:582200fc54b4dfde3ffb1c51b279cfab6861a92dc019b6302e4bafa556e91d7a
sha256:c155878be82a002e1ca93ac407b41731e0082e297407887a44cce964852950c5
sha256:3847162fdc61353ff307f196a899552582b43fb0c91e47b6d800541e1b2779e2
sha256:7a68c1d7ffbebe3f5c943a44235c239fbe68cf72d8ce12e251ced56315021c0d
sha256:7c7868d5cc2128c30112e84f2ecd7fd0afadc2815d101598a1e44e000aefdf37
sha256:b7225a3eb024a1472fe7596fdc8f5be05669f6b1ae93e9adb730f48fe0fcba52

Deleting images from OpenShift ...
IMAGE
sha256:88d9242ee0f03760babc3e01dcadf70dd312f9e04dd41dc55f120224ff810a1f



On node where docker-registry is running:
# pwd
/var/lib/openshift/openshift.local.volumes/pods/3056e61d-26b7-11e5-8319-fa163e58239e
# tree . >~/before
# du -sk .
1316	.


On client:
$ oadm prune images --keep-younger-than=0 --keep-tag-revisions=1 --confirm --registry-url='registry.jialiu.com'

Deleting registry layer blobs ...
BLOB
sha256:3f45ac862e1040ba0a9dfd28b5e06c20486b752e85016d12f87f0fb018b4f193
sha256:b7225a3eb024a1472fe7596fdc8f5be05669f6b1ae93e9adb730f48fe0fcba52
sha256:e5b1e91fc7747f5121550b1f4d605788fdaddef9d6cab3230c79362eab057ada
sha256:582200fc54b4dfde3ffb1c51b279cfab6861a92dc019b6302e4bafa556e91d7a
sha256:5880a3208e17b1185f9c933b86b5f7fbd8093c5619f9439d18213ae31a93dcbc
sha256:3847162fdc61353ff307f196a899552582b43fb0c91e47b6d800541e1b2779e2
sha256:c155878be82a002e1ca93ac407b41731e0082e297407887a44cce964852950c5
sha256:7c7868d5cc2128c30112e84f2ecd7fd0afadc2815d101598a1e44e000aefdf37
sha256:7a68c1d7ffbebe3f5c943a44235c239fbe68cf72d8ce12e251ced56315021c0d

Deleting images from OpenShift ...
IMAGE
sha256:88d9242ee0f03760babc3e01dcadf70dd312f9e04dd41dc55f120224ff810a1f


On node where docker-registry is running:
# pwd
/var/lib/openshift/openshift.local.volumes/pods/3056e61d-26b7-11e5-8319-fa163e58239e
# tree . >~/after
# du -sk .
72	.

# diff ~/before ~/after 
25,26d24
<     |                   |       |   `-- 582200fc54b4dfde3ffb1c51b279cfab6861a92dc019b6302e4bafa556e91d7a
<     |                   |       |       `-- data
82c80
< 59 directories, 20 files
---
> 58 directories, 19 files


Obviously size is becoming smaller, and some data files are delete after running prune command.

Comment 8 Johnny Liu 2015-07-13 08:07:29 UTC
In comment 7, missed openshift version, it should be openshift-3.0.1.0-0.git.167.74e2880.el7ose.x86_64

Comment 10 errata-xmlrpc 2015-08-03 20:17:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1540


Note You need to log in before you can comment on or make changes to this bug.