1487408 – Prune Deleted Layer of a Valid Image due to minimum aging

Bug 1487408 - Prune Deleted Layer of a Valid Image due to minimum aging

Summary: Prune Deleted Layer of a Valid Image due to minimum aging

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Image Registry
Sub Component:
Version:	3.5.1
Hardware:	x86_64
OS:	Linux
Priority:	urgent
Severity:	urgent
Target Milestone:	---
Target Release:	3.7.0
Assignee:	Oleg Bulatov
QA Contact:	Dongbo Yan
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1498123 1498124 1509362
TreeView+	depends on / blocked

Reported:	2017-08-31 20:14 UTC by Matthew Robson
Modified:	2021-09-09 12:34 UTC (History)
CC List:	13 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: images younger than the threshold are not added to the dependency graph. Consequence: a blob which is used by a young image and by a prunable image is deleted because it has no references in the graph. Fix: add young images to the graph and mark them as non-prunable. Result: the blob has references and is not deleted.
Clone Of:
Clones:	1498123 1498124 1509362 (view as bug list)
Environment:
Last Closed:	2017-11-28 22:08:20 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Solution)	3242401	0	None	None	None	2017-11-15 18:57:59 UTC
Red Hat Product Errata	RHSA-2017:3188	0	normal	SHIPPED_LIVE	Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update	2017-11-29 02:34:54 UTC

Description Matthew Robson 2017-08-31 20:14:26 UTC

Description of problem:

Running a prune removed a layer part of a valid image when pruning a prunable image.

Image c001b8fb0fca001953cb70fdc7d5914f242825608e9bda9e7f2b6093c3bb1bbd was validly pruned and removed layer 138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0 as well.

Aug 29:

I0829 07:01:25.789073    9257 prune.go:315] Examining image "sha256:c001b8fb0fca001953cb70fdc7d5914f242825608e9bda9e7f2b6093c3bb1bbd"
I0829 07:01:25.789084    9257 prune.go:330] Adding image "sha256:c001b8fb0fca001953cb70fdc7d5914f242825608e9bda9e7f2b6093c3bb1bbd" to graph
I0829 07:01:25.789103    9257 prune.go:341] Adding image layer "sha256:6d987f6f42797d81a318c40d442369ba3dc124883a0964d40b0c8f4f7561d913" to graph
I0829 07:01:25.789122    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0829 07:01:25.789145    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0829 07:01:25.789162    9257 prune.go:341] Adding image layer "sha256:013dbd5feb01bb1ff03719e84d434f4f44c22b433c7c210b4a5bc9d9d0d3db6b" to graph
I0829 07:01:25.789178    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0829 07:01:25.789194    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0829 07:01:25.789210    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0829 07:01:25.789227    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0829 07:01:25.789243    9257 prune.go:341] Adding image layer "sha256:9d5b23feead7df566b77b19c9e6d7b7e912d858f5ca0503ad447dff816b6fb4b" to graph
I0829 07:01:25.789259    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0829 07:01:25.789274    9257 prune.go:341] Adding image layer "sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0" to graph
I0829 07:01:25.789292    9257 prune.go:341] Adding image layer "sha256:c6d0d510aebff435403c853dd6693e8a69965de9133bb505db62fb80ea8cc837" to graph
I0829 07:01:25.789310    9257 prune.go:341] Adding image layer "sha256:55027959a1bf6801beb944116b9c872e4dcce38740af19cdd0cfb5d7ae11b5c8" to graph
I0829 07:01:25.789326    9257 prune.go:341] Adding image layer "sha256:d09178d6940d981f1df6c905db51c576052c353f712cc14d35353348cc299990" to graph
I0829 07:01:25.789343    9257 prune.go:341] Adding image layer "sha256:85525fef6f2ee79276234d528ea1f3de08bd8970186ba497b861f9ed9b5cb77b" to graph
I0829 07:01:25.789362    9257 prune.go:341] Adding image layer "sha256:c34253d823f398c59b8b4c6cec8ad66651f4b40c60818465e753694ed1e849bf" to graph
I0829 07:01:25.789381    9257 prune.go:341] Adding image layer "sha256:9374522910fc715748dace99dabeb907b1eb755275cc0095abc9b13e3e50421d" to graph
I0829 07:01:25.789398    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0829 07:01:25.789414    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0829 07:01:25.789430    9257 prune.go:341] Adding image layer "sha256:826da1a4f8ed1c7fd2671c98a1c40df8e5b092b7e8c7f0d1f0b574bedb88737e" to graph
I0829 07:01:25.789451    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0829 07:01:25.789468    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0829 07:01:25.789483    9257 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
...
...
I0829 07:01:27.144052    9257 prune.go:669] Examining image "sha256:c001b8fb0fca001953cb70fdc7d5914f242825608e9bda9e7f2b6093c3bb1bbd"
I0829 07:01:27.144062    9257 prune.go:672] Image "sha256:c001b8fb0fca001953cb70fdc7d5914f242825608e9bda9e7f2b6093c3bb1bbd" is prunable
...
...
I0829 07:01:27.494324    9257 prune.go:711] Examining ImageComponent|sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0
I0829 07:01:27.494333    9257 prune.go:713] ImageComponent|sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0 is prunable
...
I0829 07:01:27.586959    9257 prune.go:932] Pruning registry="172.30.0.2:5000", blob="sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0"

Run:
I0829 07:03:39.164668    9470 prune.go:981] Deleting image "sha256:c001b8fb0fca001953cb70fdc7d5914f242825608e9bda9e7f2b6093c3bb1bbd"
I0829 07:03:39.330426    9470 request.go:632] Throttling request took 165.691073ms, request: DELETE:https://url:8443/oapi/v1/images/sha256:c001b8fb0fca001953cb70fdc7d5914f242825608e9bda9e7f2b6093c3bb1bb
...
I0829 07:03:05.182931    9470 prune.go:932] Pruning registry="172.30.0.2:5000", blob="sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0"
I0829 07:03:05.182989    9470 prune.go:1093] Pruning registry "172.30.0.2:5000", blob "sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0"
I0829 07:03:05.183003    9470 prune.go:1050] Trying https for 172.30.0.2:5000/admin/blobs/sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0
I0829 07:03:05.183022    9470 prune.go:1016] Sending request to registry
I0829 07:03:05.187772    9470 prune.go:1062] Error with https for 172.30.0.2:5000/admin/blobs/sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0: error sending request: http: server gave HTTP response to HTTPS client
I0829 07:03:05.187795    9470 prune.go:1050] Trying http for 172.30.0.2:5000/admin/blobs/sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0
I0829 07:03:05.187813    9470 prune.go:1016] Sending request to registry

The issue is that 138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0 was also apart of image 06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3.

I believe the layer was prunable because image 06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3 was not added the graph as it was under the minimum prunable age. As such, no image predecessor reference was made to the layer.

I0829 07:01:25.206779    9257 prune.go:315] Examining image "sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3"
I0829 07:01:25.206792    9257 prune.go:326] Image "sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3" is younger than minimum pruning age, skipping (age=85h54m29.206788575s)

I0829 07:01:26.730273    9257 prune.go:365] Examining ImageStream xlrelease-poc/xl-docker-demo-xlr
I0829 07:01:26.730283    9257 prune.go:376] Adding ImageStream xlrelease-poc/xl-docker-demo-xlr to graph
I0829 07:01:26.730324    9257 prune.go:385] Unable to find image "sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3" in graph (from tag="latest", revision=0, dockerImageReference=172.50.0.2:5000/xlrelease-poc/xl-docker-demo-xlr@sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3) - skipping

I0829 07:01:26.787223    9257 prune.go:479] Examining pod xlrelease-poc/xl-docker-demo-xlr-14-4c8vg
I0829 07:01:26.787230    9257 prune.go:490] Adding pod xlrelease-poc/xl-docker-demo-xlr-14-4c8vg to graph
I0829 07:01:26.787256    9257 prune.go:504] Examining container image "172.50.0.2:5000/xlrelease-poc/xl-docker-demo-xlr@sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3"
I0829 07:01:26.787298    9257 prune.go:519] Unable to find image "sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3" in the graph - skipping

I0829 07:01:26.985453    9257 prune.go:536] Examining replication controller xlrelease-poc/xl-docker-demo-xlr-13
I0829 07:01:26.985490    9257 prune.go:504] Examining container image "172.50.0.2:5000/xlrelease-poc/xl-docker-demo-xlr@sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3"
I0829 07:01:26.985532    9257 prune.go:519] Unable to find image "sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3" in the graph - skipping

I0829 07:01:26.985552    9257 prune.go:536] Examining replication controller xlrelease-poc/xl-docker-demo-xlr-14
I0829 07:01:26.985593    9257 prune.go:504] Examining container image "172.50.0.2:5000/xlrelease-poc/xl-docker-demo-xlr@sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3"
I0829 07:01:26.985633    9257 prune.go:519] Unable to find image "sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3" in the graph - skipping

I0829 07:01:27.105356    9257 prune.go:550] Examining DeploymentConfig xlrelease-poc/xl-docker-demo-xlr
I0829 07:01:27.105384    9257 prune.go:504] Examining container image "172.50.0.2:5000/xlrelease-poc/xl-docker-demo-xlr@sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3"
I0829 07:01:27.105421    9257 prune.go:519] Unable to find image "sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3" in the graph - skipping

On August 30th, you can see it's old enough now and still referencing 138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0 as a layer.

I0830 07:01:19.756684   27103 prune.go:315] Examining image "sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3"
I0830 07:01:19.756695   27103 prune.go:330] Adding image "sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3" to graph
I0830 07:01:19.756719   27103 prune.go:341] Adding image layer "sha256:6d987f6f42797d81a318c40d442369ba3dc124883a0964d40b0c8f4f7561d913" to graph
I0830 07:01:19.756741   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.756772   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.756791   27103 prune.go:341] Adding image layer "sha256:013dbd5feb01bb1ff03719e84d434f4f44c22b433c7c210b4a5bc9d9d0d3db6b" to graph
I0830 07:01:19.756814   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.756831   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.756847   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.756864   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.756879   27103 prune.go:341] Adding image layer "sha256:9d5b23feead7df566b77b19c9e6d7b7e912d858f5ca0503ad447dff816b6fb4b" to graph
I0830 07:01:19.756898   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.756915   27103 prune.go:341] Adding image layer "sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0" to graph
I0830 07:01:19.756933   27103 prune.go:341] Adding image layer "sha256:c6d0d510aebff435403c853dd6693e8a69965de9133bb505db62fb80ea8cc837" to graph
I0830 07:01:19.756951   27103 prune.go:341] Adding image layer "sha256:58f1f028359322e79384f47f9890af0ed47bf53062d658eb43c7ca37c510a14b" to graph
I0830 07:01:19.756975   27103 prune.go:341] Adding image layer "sha256:a1f32fd2b50a5bafa6c6de5098f4cdeab152e61b0e0f5c7b2a2f4251a5156ebf" to graph
I0830 07:01:19.756994   27103 prune.go:341] Adding image layer "sha256:2f79f0ebb70e8b249550e72654b4cc4cb600c0bea30a22d212a5b203f3315af2" to graph
I0830 07:01:19.757012   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.757028   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.757044   27103 prune.go:341] Adding image layer "sha256:0e538aa21adff4ac961bbaf214f0217387449e450543af3563f7a6cb017a1dd3" to graph
I0830 07:01:19.757061   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.757078   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.757093   27103 prune.go:341] Adding image layer "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to graph
I0830 07:01:19.757108   27103 prune.go:315] Examining image "sha256:063e14a48414f6e66ec0e6570ae0f846623e08ff9416b280f560898bac7e7e63"

I0830 07:01:21.802092   27103 prune.go:867] Examining predecessor &nodes.ImageNode{Node:graph.Node{Node:233, UniqueName:"Image|/sha256:06384c77499baf997fc3ea4d15c03fd3b3086efa75606d409b794938d1c027b3"}, Image:(*api.Image)(0xc431550a30)} of image config ImageComponent|sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0
I0830 07:01:21.802113   27103 prune.go:869] Config ImageComponent|sha256:138e2afa24c729159130c68db643071d92c5a63270ac69ef2ec55215e80f66e0 has an image predecessor



Version-Release number of selected component (if applicable):
3.5.5.31

How reproducible:
Happened once, but probably very repeatable.

Steps to Reproduce:
1. Layer referenced from 2 images
2. One image is prunable while the other is under the min age
3. Layer is pruned

Actual results:
Valid layer is pruned

Expected results:
Do not prune valid layers

Additional info:

Comment 9 Oleg Bulatov 2017-10-03 13:46:47 UTC

https://github.com/openshift/origin/pull/16656

Comment 10 openshift-github-bot 2017-10-04 01:02:44 UTC

Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/917bfe7b490f182456ba05d2634fe6286fa1ff69
Test for bug 1487408

Comment 12 Dongbo Yan 2017-10-12 08:02:32 UTC

Verified
# oc version
oc v3.7.0-0.147.0
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://:8443
openshift v3.7.0-0.143.2
kubernetes v1.7.0+80709908fd

push two images reference the same image layer, one is pushed more 10min. Prune the old image via cmd
# oadm prune images --certificate-authority=ca.crt --keep-younger-than=10m --registry-url=docker-registry-default.com --confirm

Deleting registry layer blobs ...
BLOB
sha256:9f3ddeba9f24e76cfba298bfa8182fd6a59f04ef9f619c9914d6f401044f54e3
sha256:a67b0482ed5d83ab726f8c10c056dd5cb1430e7251f44c90c5a547d207999cab

Deleting images from server ...
IMAGE
sha256:3a5a0bdc0f7badf9412b0d26d20fd91e5cb015dcc868094c6d8d5b01bd23f8ae

Actual result:the new image is not pruned

move to verified

Comment 18 errata-xmlrpc 2017-11-28 22:08:20 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188

Note You need to log in before you can comment on or make changes to this bug.