Bug 1448595

Summary: oadm prune command fails with TLS issues after adding --confirm
Product: OpenShift Container Platform Reporter: Eric Jones <erjones>
Component: Image RegistryAssignee: Michal Minar <miminar>
Status: CLOSED ERRATA QA Contact: ge liu <geliu>
Severity: low Docs Contact:
Priority: low    
Version: 3.4.1CC: aos-bugs, dyan, erjones, maszulik, mfojtik, miminar, misalunk
Target Milestone: ---Keywords: Unconfirmed
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Neither documentation nor cmd help talked about insecure connections to the secured registry. Errors used to be hard to decipher when user attempted to prune secured registry with bad CA certificate. Consequence: Users had troubles running image prune against (in)secured registries. Fix: Errors are now printed with hints, cmd help has been updated, new flags have been provided to allow for insecure fall-back. Result: User can now easily enforce both secure and insecure connection. He will also be able to understand https errors and what to do when he hits them.
 Story Points: ---
Clone Of:
: 1474446 1475306 1476779 (view as bug list) Environment:
Last Closed: 2017-11-28 21:54:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1474446, 1475306, 1476779    

Description Eric Jones 2017-05-05 21:21:17 UTC 
Description of problem:
Customer sees error messages [0] when adding the --confirm flag to the command `oadm prune images --keep-tag-revisions=3 --keep-younger-than=60m`

The registry is secured so under recommendation from engineering previously we attempted to add the --certificate-authority flag as well as the --registry-url flag but to no avail [1]

[0] 
error: error communicating with registry: Get http://172.30.74.39:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

At the same time the registry logs print:
I0417 14:11:26.845937       1 logs.go:41] http: TLS handshake error from 10.1.3.1:43112: EOF
I0417 14:11:26.847464       1 logs.go:41] http: TLS handshake error from 10.1.3.1:43114: tls: first record does not look like a TLS handshake

[1] 
# oadm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm --certificate-authority=/etc/origin/master/ca.crt
error: error communicating with registry: Get http://172.30.74.39:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
# oadm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm --certificate-authority=/etc/origin/master/ca.crt --registry-url=172.30.74.39:5000
error: error communicating with registry: Get http://172.30.74.39:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

Version-Release number of selected component (if applicable):
$ oc version
oc v3.4.1.10
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO
 
Server https://osftmstr.dev.bmocm.com:8443
openshift v3.4.1.10
kubernetes v1.4.0+776c994

$ oadm version
oadm v3.4.1.10
kubernetes v1.4.0+776c994
 
Server https://osftmstr.dev.bmocm.com:8443
openshift v3.4.1.10
kubernetes v1.4.0+776c994

$ oc get dc -n default -o jsonpath='{range .items[*]}{"DC: "}{.metadata.name}{"\n Image:"}{.spec.template.spec.containers[*].image}{"\n"}{end}'
DC: docker-registry
 Image:registry.access.redhat.com/openshift3/ose-docker-registry:v3.4.1.10

Additional info:
We collected logs from the master services, docker-registry pod, and the info generated by adding --loglevel=8 to the prune command from before the attempt to add ca and url flags, will attach them shortly.
Comment 12 Michal Minar 2017-10-10 15:37:00 UTC 
Merged.
Comment 13 Dongbo Yan 2017-10-12 05:40:06 UTC 
# oc version
oc v3.7.0-0.147.0
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://:8443
openshift v3.7.0-0.143.2
kubernetes v1.7.0+80709908fd

# oadm prune images --certificate-authority=ca.crt --keep-younger-than=0 --registry-url=docker-registry-default.com --confirm 

Deleting registry layer blobs ...
BLOB
sha256:77b3ed558f11da586d2610e50069966030034e3186c8c03ec1d08db42c97ccf1
sha256:aa23e69b4bdf753cff5bdd5e6b2e1244461b859a04210d55a4a3ddf21fb4ff20
sha256:dfff00e37fce2b8e3ff94d5841ea1dc9015be0f63a8d5b2ea09b442c9c1be3ad
sha256:997108dc601097bb79f4d7a0547f36a2cabbed79877082b1f358dc081f35baee

Deleting images from server ...
IMAGE
sha256:4c6c520a0e34d14e3f08184d0fcb5cf4cb48dbee09874823ac25a661f93a4caf
sha256:ad037badcd3437a4aa0dc1312167c3fe7d6b176fe713335857b933f5b54a2f44

move to verified
Comment 17 errata-xmlrpc 2017-11-28 21:54:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188