Bug 1387762

Summary: Error Pruning image from stream: client: response is invalid json.
Product: OpenShift Container Platform Reporter: Max Whittingham <mwhittin>
Component: Image RegistryAssignee: Maciej Szulik <maszulik>
Status: CLOSED DUPLICATE QA Contact: Wei Sun <wsun>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.2.1CC: aos-bugs, mfojtik, mwhittin
Target Milestone: ---Keywords: UpcomingRelease
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-01 11:45:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1303130    

Description Max Whittingham 2016-10-21 19:29:58 UTC 
Description of problem:
During scheduled image pruning, OpenShift gets stuck with error messages and does not prune.

Version-Release number of selected component (if applicable):
tuned-profiles-atomic-openshift-node-3.2.1.17-1.git.0.6d01b60.el7.x86_64
atomic-openshift-3.2.1.17-1.git.0.6d01b60.el7.x86_64
atomic-openshift-node-3.2.1.17-1.git.0.6d01b60.el7.x86_64
atomic-openshift-master-3.2.1.17-1.git.0.6d01b60.el7.x86_64
atomic-openshift-clients-3.2.1.17-1.git.0.6d01b60.el7.x86_64
atomic-openshift-sdn-ovs-3.2.1.17-1.git.0.6d01b60.el7.x86_64


How reproducible:
Running /usr/bin/cron-openshift-pruner --image-keep-younger-than 24h --image-keep-tag-revisions 5

Actual results: Lots of the following errors
error updating image stream project-vera-dev/vera-nodejs to remove references to image sha256:0fa1724a29ed96ac71aacffdc2724718e90956d06bc7c0e5563464aef2892738: client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint.
error updating image stream project-vera-dev/vera-nodejs to remove references to image sha256:b8e8351249620a41dcec006f83abc2749bfbb82a58f127496dc8e70cb2f0b11e: client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint.
W1021 18:40:16.411907   39466 imagepruner.go:922] Unable to prune layer https://172.30.166.216:5000/v2/project-vera-dev/vera-nodejs/blobs/sha256:6557ee47e70c3c049235604151b6e6234acfce71b6c28bccf4f97557af7f4f5e, returned 404 Not Found
W1021 18:40:16.434615   39466 imagepruner.go:922] Unable to prune layer https://172.30.166.216:5000/v2/project-vera-dev/vera-nodejs/blobs/sha256:726b8179df856b0fdc1bf5256b5ee2ff9aa4b386d4d8c0b0b9087afc70c7daa9, returned 404 Not Found
error pruning image from stream: client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint.
error pruning image from stream: client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint.
Error pruning images
Traceback (most recent call last):
  File "/usr/bin/cron-openshift-pruner", line 222, in <module>
    OSPruner.main()
  File "/usr/bin/cron-openshift-pruner", line 218, in main
    raise Exception("Error during pruning")
Exception: Error during pruning
Comment 1 Maciej Szulik 2016-11-02 12:08:41 UTC 
What is /usr/bin/cron-openshift-pruner and where did you get it from? AFAIC all pruning operations should directly invoke oadm prune images which in turn connects to OpenShift API to remove images and imagestreams, and connects to internal registry to remove the images from there as well. We don't connect directly to etcd to remove any data from there. If you're running oadm prune images under the hood, can you give me the result of this command with --loglevel=7?
Comment 2 Max Whittingham 2016-11-30 22:45:40 UTC 
It looks like this bug is duplicated by 1377739 and can be closed in favor of that bug.
Comment 3 Maciej Szulik 2016-12-01 11:45:56 UTC 
Per reporter justification closing as duplicate.

*** This bug has been marked as a duplicate of bug 1377739 ***