Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1861102

Summary: OCP 4.5.3 - Disconnected IPI on OSP13 with https_proxy & Etcd Encryption - cluster operator kube-controller-manager is degraded during upgrade
Product: OpenShift Container Platform Reporter: Walid A. <wabouham>
Component: InstallerAssignee: Eric Duen <eduen>
Installer sub component: OpenShift on OpenStack QA Contact: David Sanz <dsanzmor>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: high    
Priority: unspecified CC: aos-bugs, eduen, kewang, knarra, lmohanty, mfojtik, mifiedle, pprinett, tnozicka, wking, xxia
Version: 4.5Keywords: UpcomingSprint, Upgrades
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-20 14:27:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Walid A. 2020-07-27 20:13:05 UTC
Description of problem:
This is observed on a Disconnected IPI OCP cluster on OSP13 with https_proxy & Etcd Encryption on.  While running the QE Upgrade CI to upgrade OCP cluster from version 4.5.3 to 4.5.0-0.nightly-2020-07-25-031342, cluster-operator kube-api-controller has degraded causing the upgrade to fail

Version-Release number of selected component (if applicable):
ocp v4.5.3
kubernetes v1.18.3+3107688
Red Hat Enterprise Linux CoreOS 45.82.202007171855-0 (Ootpa)   4.18.0-193.13.2.el8_2.x86_64
cri-o://1.18.3-4.rhaos4.5.gitb5e3b15.el8

How reproducible:
Seen once

Steps to Reproduce:
1. Install a Disconnected IPI OCP cluster on OSP13 with https_proxy & Etcd Encryption on
2. After successful install, patch clusterversion config
3. oc adm upgrade --to=4.5.0-0.nightly-2020-07-25-031342

Actual results:
Upgrade fails.  oc get clusterversion shows:
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.3     True        False         70m     Error while reconciling 4.5.3: the cluster operator kube-controller-manager is degraded

kube-controller-manager is degraded:             4.5.3     True        False         True       3h14m


Expected results:
Upgrade to be successful and all cluster operators not degraded

Additional info:
Link to must-gather and oc commands in next private comment

Comment 2 RamaKasturi 2020-07-28 08:34:32 UTC
I tried a manual upgrade today on the same profile and i hit the bug https://bugzilla.redhat.com/show_bug.cgi?id=1858498. Currently we need a way to resolve the conflict during upgrade or before upgrading so that upgrade is not blocked totally.

upgrade from 4.5.3 -> 4.5.0-0.nightly-2020-07-25-031342

[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.3     True        True          109m    Unable to apply 4.5.0-0.nightly-2020-07-25-031342: the cluster operator kube-controller-manager is degraded


I have a live cluster available in case some one would want to take a look, thanks !!

Comment 3 Tomáš Nožička 2020-07-28 13:06:30 UTC
Looking at must-gather, this is the haproxy issue from https://bugzilla.redhat.com/show_bug.cgi?id=1858498. Sending it to the team owning that component.

```
  - containerID: cri-o://42c813049c458ff05ab5bcff1658bc29b93bd82d08cf9b4a7694bb75156833ec
    image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:74e4c487ba8ecc2c94f70e13242d6dc35791dcdcee5cfb2f30540535ea6f492f
    imageID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:74e4c487ba8ecc2c94f70e13242d6dc35791dcdcee5cfb2f30540535ea6f492f
    lastState:
      terminated:
        containerID: cri-o://42c813049c458ff05ab5bcff1658bc29b93bd82d08cf9b4a7694bb75156833ec
        exitCode: 124
        finishedAt: "2020-07-27T10:51:24Z"
        message: "43                [::ffff:127.0.0.1]:38836              \nESTAB
          \     0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:39872
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:60290
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:40836
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:40876
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:50746
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:55510
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:42266
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:57870
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:46338
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:38192
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:52086
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:58534
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:40574
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:47958
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:49696
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:43508
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:34546
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:52496
          \             \nESTAB      0      0       [::ffff:127.0.0.1]:9443                [::ffff:127.0.0.1]:59852
          \             ' ']'\n+ sleep 1\n"
        reason: Error
        startedAt: "2020-07-27T10:48:24Z"
    name: kube-controller-manager-recovery-controller
    ready: false
    restartCount: 19
    started: false
    state:
      waiting:
        message: back-off 5m0s restarting failed container=kube-controller-manager-recovery-controller
          pod=kube-controller-manager-ugdci27151320-4smpl-master-0_openshift-kube-controller-manager(9c97575ab28e8923f54d68886facc852)
        reason: CrashLoopBackOff
```

Comment 10 RamaKasturi 2020-08-06 15:32:52 UTC
Hi Walid,
 
    I performed the workaround as per comment 7 and could see that kube-controller-manager came up with out any issues. 

    [ramakasturinarra@dhcp35-60 ~]$ oc exec -n openshift-openstack-infra haproxy-kasturi-upg2-t55qp-master-0 -- grep 9443 /etc/haproxy/haproxy.cfg
Defaulting container name to haproxy.
Use 'oc describe pod/haproxy-kasturi-upg2-t55qp-master-0 -n openshift-openstack-infra' to see all of the containers in this pod.
command terminated with exit code 1
[ramakasturinarra@dhcp35-60 ~]$ oc exec -n openshift-openstack-infra haproxy-kasturi-upg2-t55qp-master-0 -- grep 9445/etc/haproxy/haproxy.cfg
Defaulting container name to haproxy.
Use 'oc describe pod/haproxy-kasturi-upg2-t55qp-master-0 -n openshift-openstack-infra' to see all of the containers in this pod.
command terminated with exit code 1
[ramakasturinarra@dhcp35-60 ~]$ oc exec -n openshift-openstack-infra haproxy-kasturi-upg2-t55qp-master-0 -- grep 9445 /etc/haproxy/haproxy.cfg
Defaulting container name to haproxy.
Use 'oc describe pod/haproxy-kasturi-upg2-t55qp-master-0 -n openshift-openstack-infra' to see all of the containers in this pod.
  bind :::9445 v4v6
[ramakasturinarra@dhcp35-60 ~]$ oc exec -n openshift-openstack-infra haproxy-kasturi-upg2-t55qp-master-1 -- grep 9445 /etc/haproxy/haproxy.cfg
Defaulting container name to haproxy.
Use 'oc describe pod/haproxy-kasturi-upg2-t55qp-master-1 -n openshift-openstack-infra' to see all of the containers in this pod.
  bind :::9445 v4v6
[ramakasturinarra@dhcp35-60 ~]$ oc exec -n openshift-openstack-infra haproxy-kasturi-upg2-t55qp-master-2 -- grep 9445 /etc/haproxy/haproxy.cfg
Defaulting container name to haproxy.
Use 'oc describe pod/haproxy-kasturi-upg2-t55qp-master-2 -n openshift-openstack-infra' to see all of the containers in this pod.
  bind :::9445 v4v6

After that tried upgrading from 4.5.3 -> 4.5.0-0.nightly-2020-08-06-062632 and the upgrade was successful with respect to kube-controller-manager but hit issue https://bugzilla.redhat.com/show_bug.cgi?id=1826150.

Thanks
kasturi

Comment 11 Pierre Prinetti 2020-08-20 14:27:15 UTC
Closing. Thank you!