Bug 1543511 – CRI-O runtime - image pruning fails

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1543511 - CRI-O runtime - image pruning fails

Summary:	CRI-O runtime - image pruning fails

Status:	CLOSED ERRATA

Aliases:	None

Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Image Registry (Show other bugs)
Sub Component:
Version:	3.9.0
Hardware:	Unspecified Unspecified

Priority	unspecified Severity medium
Target Milestone:	---
Target Release:	3.9.0
Assigned To:	Alexey Gladkov
QA Contact:	Dongbo Yan
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2018-02-08 10:36 EST by Vikas Laad
Modified:	2018-03-28 10:27 EDT (History)
CC List:	4 users (show)

See Also:
Fixed In Version:
Doc Type:	Enhancement
Doc Text:	Feature: Make image pruner tolerate since to empty docker image reference. Reason: In the new version of kube, a space is used to indicate that the field is empty, but is flooded. Result: The image pruner ignore the field containing space.
Story Points:	---
Clone Of:
Environment:
Last Closed:	2018-03-28 10:27:28 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2018:0489	None	None	None	2018-03-28 10:27 EDT

Groups: None (edit)

Description Vikas Laad 2018-02-08 10:36:59 EST

Description of problem:
Image pruning is failing during reliability test run, these tests run over a period of 2 weeks and keeps creating/building/scaling/accessing quickstart apps.

root@ip-172-31-9-217: ~ # oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm
Failed to build graph!

The following objects have invalid references:

  DeploymentConfig[nodejs-mongodb-u-820-294-837-301/nodejs-mongodb-example]: invalid docker image reference " ": invalid reference format

Either fix the references or delete the objects to make the pruner proceed.
error: failed to build graph - no changes made

I am trying to do pruning with a user which has following role
system:image-pruner                                                   /system:image-pruner                                                   redhat                                                                          default/registry                                                                                                   

I also see following Warnings on nodes
  Warning  ImageGCFailed          8m (x93 over 7h)   kubelet, ip-172-31-11-219.us-west-2.compute.internal  (combined from similar events): wanted to free 6071304192 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Image used by 6c28f3ff59333616c4d54998e9db5ca60e8b0b92600202bcbfe980916dd398dd: image is in use by a container, rpc error: code = Unknown desc = Image used by b2f15b6484145363112b005f5597f4cf826cc09dd8ad3178495241e84dfa351c: image is in use by a container, rpc error: code = Unknown desc = Image used by a37256b6db66f6e8351d97633f6cf6c3478c742e0312e44020a06c3d5757cfc0: image is in use by a container, rpc error: code = Unknown desc = Image used by 552f10aa6cd9e06c1940263b7f70d7d673e52808acefb11448f84a1112457346: image is in use by a container, rpc error: code = Unknown desc = Image used by e9baeaf58fc0285ba1c82f8e3fbc2ebc9d193cb0c245c28a362d2c02466eb716: image is in use by a container, rpc error: code = Unknown desc = Image used by 23627270dc49a342ce298dff6f84d96b8fd922f01cfe2b371ef73505ce261544: image is in use by a container, rpc error: code = Unknown desc = Image used by 6c56ad0acb0dd0064386c6485f686dee0ad62968e89bc918577a4fbd0b6201f2: image is in use by a container, rpc error: code = Unknown desc = Image used by 22b668ee3a26f2efff038a02f059b3263501a2a61726c92c166d1f8c8c003cdc: image is in use by a container, rpc error: code = Unknown desc = Image used by 1a82132c2a9d7846130844ea3841cb4f6a20acdddb2b7422ca3abea090af0df7: image is in use by a container, rpc error: code = Unknown desc = Image used by 4e811f0ea557fa207c3ad184f9aa7e4809f2d32f0194cf120f3cc5e6c8441980: image is in use by a container, rpc error: code = Unknown desc = Image used by 711ad77cb87b4d3c0826b439ccd6486457b9b4262daeb9f14bfdd1ecc1c973a8: image is in use by a container, rpc error: code = Unknown desc = Image used by 3bf7967be8ef8f90ad877acfcf604cc8f559c8b079e496a096560051326617fe: image is in use by a container]


Version-Release number of selected component (if applicable):
openshift v3.9.0-0.36.0
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.8

How reproducible:


Steps to Reproduce:
1. create/build/delete quickstart apps for long time
2. provide image-pruner role to a user and login with that user
3. oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm

Actual results:
Errors

Expected results:
Pruning should work

Additional info:
Attaching master and node logs.

Comment 4 Ben Parees 2018-02-08 10:48:10 EST

ImageGC warnings are unrelated to registry pruning, please open a separate issue for that.

The registry pruning is unrelated to CRIO.  The failure you're getting:

DeploymentConfig[nodejs-mongodb-u-820-294-837-301/nodejs-mongodb-example]: invalid docker image reference " ": invalid reference format

Is something I guess we need to make our pruner tolerate since we explicitly tell people to use a docker image reference value of " " in order to work around some limitations of the API.  I'm surprised you're the first person to hit this.

Comment 5 Alexey Gladkov 2018-02-12 08:35:35 EST

Can you test this fix https://github.com/openshift/origin/pull/18570 ?

Comment 6 Vikas Laad 2018-02-12 09:20:20 EST

Alexey, I wont be able to test it until its available in puddle.

Comment 8 Vikas Laad 2018-02-26 15:05:08 EST

Started the reliability tests run.

Comment 9 Vikas Laad 2018-03-06 09:00:31 EST

Verified on following version

openshift v3.9.0-0.53.0
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.8

Comment 12 errata-xmlrpc 2018-03-28 10:27:28 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489

Note You need to log in before you can comment on or make changes to this bug.