Bug 1448595 - oadm prune command fails with TLS issues after adding --confirm
Summary: oadm prune command fails with TLS issues after adding --confirm
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.4.1
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 3.7.0
Assignee: Michal Minar
QA Contact: ge liu
URL:
Whiteboard:
Depends On:
Blocks: 1474446 1475306 1476779
TreeView+ depends on / blocked
 
Reported: 2017-05-05 21:21 UTC by Eric Jones
Modified: 2020-09-10 10:32 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Neither documentation nor cmd help talked about insecure connections to the secured registry. Errors used to be hard to decipher when user attempted to prune secured registry with bad CA certificate. Consequence: Users had troubles running image prune against (in)secured registries. Fix: Errors are now printed with hints, cmd help has been updated, new flags have been provided to allow for insecure fall-back. Result: User can now easily enforce both secure and insecure connection. He will also be able to understand https errors and what to do when he hits them.
Clone Of:
: 1474446 1475306 1476779 (view as bug list)
Environment:
Last Closed: 2017-11-28 21:54:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Eric Jones 2017-05-05 21:21:17 UTC
Description of problem:
Customer sees error messages [0] when adding the --confirm flag to the command `oadm prune images --keep-tag-revisions=3 --keep-younger-than=60m`

The registry is secured so under recommendation from engineering previously we attempted to add the --certificate-authority flag as well as the --registry-url flag but to no avail [1]

[0] 
error: error communicating with registry: Get http://172.30.74.39:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

At the same time the registry logs print:
I0417 14:11:26.845937       1 logs.go:41] http: TLS handshake error from 10.1.3.1:43112: EOF
I0417 14:11:26.847464       1 logs.go:41] http: TLS handshake error from 10.1.3.1:43114: tls: first record does not look like a TLS handshake

[1] 
# oadm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm --certificate-authority=/etc/origin/master/ca.crt
error: error communicating with registry: Get http://172.30.74.39:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
# oadm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm --certificate-authority=/etc/origin/master/ca.crt --registry-url=172.30.74.39:5000
error: error communicating with registry: Get http://172.30.74.39:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

Version-Release number of selected component (if applicable):
$ oc version
oc v3.4.1.10
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO
 
Server https://osftmstr.dev.bmocm.com:8443
openshift v3.4.1.10
kubernetes v1.4.0+776c994

$ oadm version
oadm v3.4.1.10
kubernetes v1.4.0+776c994
 
Server https://osftmstr.dev.bmocm.com:8443
openshift v3.4.1.10
kubernetes v1.4.0+776c994

$ oc get dc -n default -o jsonpath='{range .items[*]}{"DC: "}{.metadata.name}{"\n Image:"}{.spec.template.spec.containers[*].image}{"\n"}{end}'
DC: docker-registry
 Image:registry.access.redhat.com/openshift3/ose-docker-registry:v3.4.1.10

Additional info:
We collected logs from the master services, docker-registry pod, and the info generated by adding --loglevel=8 to the prune command from before the attempt to add ca and url flags, will attach them shortly.

Comment 12 Michal Minar 2017-10-10 15:37:00 UTC
Merged.

Comment 13 Dongbo Yan 2017-10-12 05:40:06 UTC
# oc version
oc v3.7.0-0.147.0
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://:8443
openshift v3.7.0-0.143.2
kubernetes v1.7.0+80709908fd

# oadm prune images --certificate-authority=ca.crt --keep-younger-than=0 --registry-url=docker-registry-default.com --confirm 

Deleting registry layer blobs ...
BLOB
sha256:77b3ed558f11da586d2610e50069966030034e3186c8c03ec1d08db42c97ccf1
sha256:aa23e69b4bdf753cff5bdd5e6b2e1244461b859a04210d55a4a3ddf21fb4ff20
sha256:dfff00e37fce2b8e3ff94d5841ea1dc9015be0f63a8d5b2ea09b442c9c1be3ad
sha256:997108dc601097bb79f4d7a0547f36a2cabbed79877082b1f358dc081f35baee

Deleting images from server ...
IMAGE
sha256:4c6c520a0e34d14e3f08184d0fcb5cf4cb48dbee09874823ac25a661f93a4caf
sha256:ad037badcd3437a4aa0dc1312167c3fe7d6b176fe713335857b933f5b54a2f44

move to verified

Comment 17 errata-xmlrpc 2017-11-28 21:54:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.