Bug 1861102
| Summary: | OCP 4.5.3 - Disconnected IPI on OSP13 with https_proxy & Etcd Encryption - cluster operator kube-controller-manager is degraded during upgrade | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Walid A. <wabouham> |
| Component: | Installer | Assignee: | Eric Duen <eduen> |
| Installer sub component: | OpenShift on OpenStack | QA Contact: | David Sanz <dsanzmor> |
| Status: | CLOSED CURRENTRELEASE | Docs Contact: | |
| Severity: | high | ||
| Priority: | unspecified | CC: | aos-bugs, eduen, kewang, knarra, lmohanty, mfojtik, mifiedle, pprinett, tnozicka, wking, xxia |
| Version: | 4.5 | Keywords: | UpcomingSprint, Upgrades |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-08-20 14:27:15 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Walid A.
2020-07-27 20:13:05 UTC
I tried a manual upgrade today on the same profile and i hit the bug https://bugzilla.redhat.com/show_bug.cgi?id=1858498. Currently we need a way to resolve the conflict during upgrade or before upgrading so that upgrade is not blocked totally. upgrade from 4.5.3 -> 4.5.0-0.nightly-2020-07-25-031342 [ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.3 True True 109m Unable to apply 4.5.0-0.nightly-2020-07-25-031342: the cluster operator kube-controller-manager is degraded I have a live cluster available in case some one would want to take a look, thanks !! Looking at must-gather, this is the haproxy issue from https://bugzilla.redhat.com/show_bug.cgi?id=1858498. Sending it to the team owning that component. ``` - containerID: cri-o://42c813049c458ff05ab5bcff1658bc29b93bd82d08cf9b4a7694bb75156833ec image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:74e4c487ba8ecc2c94f70e13242d6dc35791dcdcee5cfb2f30540535ea6f492f imageID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:74e4c487ba8ecc2c94f70e13242d6dc35791dcdcee5cfb2f30540535ea6f492f lastState: terminated: containerID: cri-o://42c813049c458ff05ab5bcff1658bc29b93bd82d08cf9b4a7694bb75156833ec exitCode: 124 finishedAt: "2020-07-27T10:51:24Z" message: "43 [::ffff:127.0.0.1]:38836 \nESTAB \ 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:39872 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:60290 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:40836 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:40876 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:50746 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:55510 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:42266 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:57870 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:46338 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:38192 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:52086 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:58534 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:40574 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:47958 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:49696 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:43508 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:34546 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:52496 \ \nESTAB 0 0 [::ffff:127.0.0.1]:9443 [::ffff:127.0.0.1]:59852 \ ' ']'\n+ sleep 1\n" reason: Error startedAt: "2020-07-27T10:48:24Z" name: kube-controller-manager-recovery-controller ready: false restartCount: 19 started: false state: waiting: message: back-off 5m0s restarting failed container=kube-controller-manager-recovery-controller pod=kube-controller-manager-ugdci27151320-4smpl-master-0_openshift-kube-controller-manager(9c97575ab28e8923f54d68886facc852) reason: CrashLoopBackOff ``` Hi Walid,
I performed the workaround as per comment 7 and could see that kube-controller-manager came up with out any issues.
[ramakasturinarra@dhcp35-60 ~]$ oc exec -n openshift-openstack-infra haproxy-kasturi-upg2-t55qp-master-0 -- grep 9443 /etc/haproxy/haproxy.cfg
Defaulting container name to haproxy.
Use 'oc describe pod/haproxy-kasturi-upg2-t55qp-master-0 -n openshift-openstack-infra' to see all of the containers in this pod.
command terminated with exit code 1
[ramakasturinarra@dhcp35-60 ~]$ oc exec -n openshift-openstack-infra haproxy-kasturi-upg2-t55qp-master-0 -- grep 9445/etc/haproxy/haproxy.cfg
Defaulting container name to haproxy.
Use 'oc describe pod/haproxy-kasturi-upg2-t55qp-master-0 -n openshift-openstack-infra' to see all of the containers in this pod.
command terminated with exit code 1
[ramakasturinarra@dhcp35-60 ~]$ oc exec -n openshift-openstack-infra haproxy-kasturi-upg2-t55qp-master-0 -- grep 9445 /etc/haproxy/haproxy.cfg
Defaulting container name to haproxy.
Use 'oc describe pod/haproxy-kasturi-upg2-t55qp-master-0 -n openshift-openstack-infra' to see all of the containers in this pod.
bind :::9445 v4v6
[ramakasturinarra@dhcp35-60 ~]$ oc exec -n openshift-openstack-infra haproxy-kasturi-upg2-t55qp-master-1 -- grep 9445 /etc/haproxy/haproxy.cfg
Defaulting container name to haproxy.
Use 'oc describe pod/haproxy-kasturi-upg2-t55qp-master-1 -n openshift-openstack-infra' to see all of the containers in this pod.
bind :::9445 v4v6
[ramakasturinarra@dhcp35-60 ~]$ oc exec -n openshift-openstack-infra haproxy-kasturi-upg2-t55qp-master-2 -- grep 9445 /etc/haproxy/haproxy.cfg
Defaulting container name to haproxy.
Use 'oc describe pod/haproxy-kasturi-upg2-t55qp-master-2 -n openshift-openstack-infra' to see all of the containers in this pod.
bind :::9445 v4v6
After that tried upgrading from 4.5.3 -> 4.5.0-0.nightly-2020-08-06-062632 and the upgrade was successful with respect to kube-controller-manager but hit issue https://bugzilla.redhat.com/show_bug.cgi?id=1826150.
Thanks
kasturi
Closing. Thank you! |