Description of problem: During scheduled image pruning, OpenShift gets stuck with error messages and does not prune. Version-Release number of selected component (if applicable): tuned-profiles-atomic-openshift-node-3.2.1.17-1.git.0.6d01b60.el7.x86_64 atomic-openshift-3.2.1.17-1.git.0.6d01b60.el7.x86_64 atomic-openshift-node-3.2.1.17-1.git.0.6d01b60.el7.x86_64 atomic-openshift-master-3.2.1.17-1.git.0.6d01b60.el7.x86_64 atomic-openshift-clients-3.2.1.17-1.git.0.6d01b60.el7.x86_64 atomic-openshift-sdn-ovs-3.2.1.17-1.git.0.6d01b60.el7.x86_64 How reproducible: Running /usr/bin/cron-openshift-pruner --image-keep-younger-than 24h --image-keep-tag-revisions 5 Actual results: Lots of the following errors error updating image stream project-vera-dev/vera-nodejs to remove references to image sha256:0fa1724a29ed96ac71aacffdc2724718e90956d06bc7c0e5563464aef2892738: client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint. error updating image stream project-vera-dev/vera-nodejs to remove references to image sha256:b8e8351249620a41dcec006f83abc2749bfbb82a58f127496dc8e70cb2f0b11e: client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint. W1021 18:40:16.411907 39466 imagepruner.go:922] Unable to prune layer https://172.30.166.216:5000/v2/project-vera-dev/vera-nodejs/blobs/sha256:6557ee47e70c3c049235604151b6e6234acfce71b6c28bccf4f97557af7f4f5e, returned 404 Not Found W1021 18:40:16.434615 39466 imagepruner.go:922] Unable to prune layer https://172.30.166.216:5000/v2/project-vera-dev/vera-nodejs/blobs/sha256:726b8179df856b0fdc1bf5256b5ee2ff9aa4b386d4d8c0b0b9087afc70c7daa9, returned 404 Not Found error pruning image from stream: client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint. error pruning image from stream: client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint. Error pruning images Traceback (most recent call last): File "/usr/bin/cron-openshift-pruner", line 222, in <module> OSPruner.main() File "/usr/bin/cron-openshift-pruner", line 218, in main raise Exception("Error during pruning") Exception: Error during pruning
What is /usr/bin/cron-openshift-pruner and where did you get it from? AFAIC all pruning operations should directly invoke oadm prune images which in turn connects to OpenShift API to remove images and imagestreams, and connects to internal registry to remove the images from there as well. We don't connect directly to etcd to remove any data from there. If you're running oadm prune images under the hood, can you give me the result of this command with --loglevel=7?
It looks like this bug is duplicated by 1377739 and can be closed in favor of that bug.
Per reporter justification closing as duplicate. *** This bug has been marked as a duplicate of bug 1377739 ***