Bug 1599428
| Summary: | Need a better error message when oc commands timeout/fail during storage upgrade | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mike Fiedler <mifiedle> | ||||||||
| Component: | oc | Assignee: | Juan Vallejo <jvallejo> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Mike Fiedler <mifiedle> | ||||||||
| Severity: | medium | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | 3.10.0 | CC: | aos-bugs, deads, jokerman, mifiedle, mmccomas, vlaad | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | 3.11.0 | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2018-10-11 07:21:36 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | 1616840 | ||||||||||
| Bug Blocks: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
Mike Fiedler
2018-07-09 19:03:18 UTC
Created attachment 1457853 [details]
loglevel=8 output
Does the oc binary still give the "CancelRequest not implemented" error if used against a different cluster than the one that was just upgraded? Also, can you confirm that the version of `oc` that you're getting this error on is 3.10? Based on your attachment, the error message appears to be originating from the UserAgent round-tripper's CancelRequest method [2]. It could be that, due to altered config during the upgrade process, the round-tripper being used here [3] does not implement a CancelRequest method, causing the error message seen (and the delay in executing commands). Adding David in case he can provide more information as well. 1. https://github.com/openshift/openshift-ansible/blob/release-3.10/playbooks/openshift-master/private/upgrade.yml#L69 2. https://github.com/openshift/origin/blob/release-3.10/vendor/k8s.io/kubernetes/staging/src/k8s.io/client-go/transport/round_trippers.go#L169 3. https://github.com/openshift/origin/blob/release-3.10/vendor/k8s.io/kubernetes/staging/src/k8s.io/client-go/transport/round_trippers.go#L37 1. Using the client against a different cluster is successful. To be clear, the cluster where the error occurs is in the middle of an upgrade, it has not yet been fully upgraded. 2. At the time the error occurs, the client is 3.10 and the api servers are still 3.9: root@ip-172-31-20-191: ~ # oc get pods E0717 20:02:00.400949 8386 round_trippers.go:169] CancelRequest not implemented ^C root@ip-172-31-20-191: ~ # oc version oc v3.10.18 kubernetes v1.10.0+b81c8f8 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://ip-172-31-20-191.us-west-2.compute.internal:8443 openshift v3.9.33 kubernetes v1.9.1+a0ce1bc657 Mike, could you provide --loglevel 8 output? The reason why this is happening is most likely because a wrapped round tripper does not implement the CancelRequest method. Created attachment 1472760 [details] oc output with loglevel=8 oc v3.10.27 kubernetes v1.10.0+b81c8f8 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://ip-172-31-61-1.us-west-2.compute.internal:8443 openshift v3.9.40 kubernetes v1.9.1+a0ce1bc657 Created attachment 1473652 [details]
patched oc log
Moving to MODIFIED until a build is ready for QE Mike, FYI, the PR is merged in OCP new puddles >= v3.11.0-0.12.0, thx Verified on 3.11.0-0.24.0 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652 |