Description of problem: Image pruning is failing during reliability test run, these tests run over a period of 2 weeks and keeps creating/building/scaling/accessing quickstart apps. root@ip-172-31-9-217: ~ # oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm Failed to build graph! The following objects have invalid references: DeploymentConfig[nodejs-mongodb-u-820-294-837-301/nodejs-mongodb-example]: invalid docker image reference " ": invalid reference format Either fix the references or delete the objects to make the pruner proceed. error: failed to build graph - no changes made I am trying to do pruning with a user which has following role system:image-pruner /system:image-pruner redhat default/registry I also see following Warnings on nodes Warning ImageGCFailed 8m (x93 over 7h) kubelet, ip-172-31-11-219.us-west-2.compute.internal (combined from similar events): wanted to free 6071304192 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Image used by 6c28f3ff59333616c4d54998e9db5ca60e8b0b92600202bcbfe980916dd398dd: image is in use by a container, rpc error: code = Unknown desc = Image used by b2f15b6484145363112b005f5597f4cf826cc09dd8ad3178495241e84dfa351c: image is in use by a container, rpc error: code = Unknown desc = Image used by a37256b6db66f6e8351d97633f6cf6c3478c742e0312e44020a06c3d5757cfc0: image is in use by a container, rpc error: code = Unknown desc = Image used by 552f10aa6cd9e06c1940263b7f70d7d673e52808acefb11448f84a1112457346: image is in use by a container, rpc error: code = Unknown desc = Image used by e9baeaf58fc0285ba1c82f8e3fbc2ebc9d193cb0c245c28a362d2c02466eb716: image is in use by a container, rpc error: code = Unknown desc = Image used by 23627270dc49a342ce298dff6f84d96b8fd922f01cfe2b371ef73505ce261544: image is in use by a container, rpc error: code = Unknown desc = Image used by 6c56ad0acb0dd0064386c6485f686dee0ad62968e89bc918577a4fbd0b6201f2: image is in use by a container, rpc error: code = Unknown desc = Image used by 22b668ee3a26f2efff038a02f059b3263501a2a61726c92c166d1f8c8c003cdc: image is in use by a container, rpc error: code = Unknown desc = Image used by 1a82132c2a9d7846130844ea3841cb4f6a20acdddb2b7422ca3abea090af0df7: image is in use by a container, rpc error: code = Unknown desc = Image used by 4e811f0ea557fa207c3ad184f9aa7e4809f2d32f0194cf120f3cc5e6c8441980: image is in use by a container, rpc error: code = Unknown desc = Image used by 711ad77cb87b4d3c0826b439ccd6486457b9b4262daeb9f14bfdd1ecc1c973a8: image is in use by a container, rpc error: code = Unknown desc = Image used by 3bf7967be8ef8f90ad877acfcf604cc8f559c8b079e496a096560051326617fe: image is in use by a container] Version-Release number of selected component (if applicable): openshift v3.9.0-0.36.0 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.8 How reproducible: Steps to Reproduce: 1. create/build/delete quickstart apps for long time 2. provide image-pruner role to a user and login with that user 3. oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm Actual results: Errors Expected results: Pruning should work Additional info: Attaching master and node logs.
ImageGC warnings are unrelated to registry pruning, please open a separate issue for that. The registry pruning is unrelated to CRIO. The failure you're getting: DeploymentConfig[nodejs-mongodb-u-820-294-837-301/nodejs-mongodb-example]: invalid docker image reference " ": invalid reference format Is something I guess we need to make our pruner tolerate since we explicitly tell people to use a docker image reference value of " " in order to work around some limitations of the API. I'm surprised you're the first person to hit this.
Can you test this fix https://github.com/openshift/origin/pull/18570 ?
Alexey, I wont be able to test it until its available in puddle.
Started the reliability tests run.
Verified on following version openshift v3.9.0-0.53.0 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.8
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489