Bug 1847181

Summary: Support for amphora to ovn-octavia upgrades
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: NetworkingAssignee: Luis Tomas Bolivar <ltomasbo>
Networking sub component: kuryr QA Contact: GenadiC <gcheresh>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: high CC: mbridges, rlobillo
Version: 4.4   
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Release Note
Doc Text:
Now it is possible to upgrade OpenShift clusters that have been deployed before OVN Octavia driver was made available on the OpenStack side. Only action needed is to ensure ovn octavia driver is available and then trigger the migration by removing the next annotation into the kuryr-config ConfigMap at openshift-kuryr namespace: - networkoperator.openshift.io/kuryr-octavia-provider: default Note this will have an impact on the connectivity that goes through those services as the loadbalancer that backs them up needs to be recreated (remove amphora load balancer and create ovn load balancer). Thus a few seconds of downtime is expected.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-13 17:44:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1846862, 1847926    
Bug Blocks:    
Attachments:
Description Flags
kuryr-controller logs during octavia upgrade none

Description OpenShift BugZilla Robot 2020-06-15 20:27:11 UTC 
+++ This bug was initially created as a clone of Bug #1846862 +++

If ovn-octavia driver is available, there should be an option to use it on existing deployments so that the resource consumption problem of having one amphora VM per service is avoided
Comment 3 rlobillo 2020-06-17 14:38:38 UTC 
Blocked until https://bugzilla.redhat.com/show_bug.cgi?id=1847926 is fixed.

Verifications steps, for the record:

1. On OSP16 with OVN, disable ovn-octavia on all the controllers:
  - modifying octavia.conf
  - restarting octavia containers
  - check openstack loadbalancer provider list.
2. Deploy OCP4.5. Check amphoras are used.
3. Enable ovn-octavia on all the controllers (undo step 1)
4. Follow https://github.com/openshift/openshift-docs/pull/22878/files.
Comment 4 rlobillo 2020-06-24 16:32:29 UTC 
Created attachment 1698636 [details]
kuryr-controller logs during octavia upgrade
Comment 5 rlobillo 2020-06-24 16:35:58 UTC 
Verified on OCP4.5.0-0.nightly-2020-06-24-055959 over OSP16 (RHOS_TRUNK-16.0-RHEL-8-20200513.n.1) with OVN.

# 1. disable ovn-octavia on all the controllers:

Modify octavia.conf with enabled_provider_drivers=amphora: The Octavia Amphora driver.,octavia: Deprecated alias of the Octavia Amphora driver.


[root@controller-0 ~]# vim /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf
-bash: vim: command not found
[root@controller-0 ~]# vi /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf
[root@controller-0 ~]# vi /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf
[root@controller-0 ~]# docker restart octavia_worker
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
7b466c70d77806a31b7a7dc9877fd9560a7a624d15bfd8d4fc4c5b70ad10c424
[root@controller-0 ~]# docker restart octavia_housekeeping
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
bce35cb4431d9dfa64a5e9a90ebdc6a68cf700ea2dabf8ddcb87d5a81804629d
[root@controller-0 ~]# docker restart octavia_api
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
4337252d0f34cc505dbb22a86b8651cc558fb6da4ff87d8d47e72de4371dba78
[root@controller-0 ~]# docker restart octavia_health_manager
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
c7b64a7a7e147336e6b4a09a8ae45ab813f99f5e01a7cf48d1a1a244660ce2b6

[root@controller-1 ~]# vi /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf
[root@controller-1 ~]# docker restart octavia_worker && docker restart octavia_housekeeping && docker restart octavia_api && docker restart octavia_health_manager
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
3d2987c0aa65931958e88e7f405747559cc8bb3f695aba312506a14850c1dbe5
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
e68f833fd9351f330cefbeade00cfba66a8c332ab319f436fcb60d8a4c927a9a
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
015385bbd8efc2d7057e5e1e0370f847b01831d05033b7715dd521e5ef6b6cf2
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
94de52ab929c32236a1b64b22fb30090b481763c0cfd66317d7c27142d8f18d2

[root@controller-2 ~]# vi /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf
[root@controller-2 ~]# docker restart octavia_worker && docker restart octavia_housekeeping && docker restart octavia_api && docker restart octavia_health_manager
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
036fce7d254a7350c0795b6b4e5f6344e03ccb49da405213e452898d71748ab8
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
8f87a6ca3c0d84dea34002b5a67da61d443b6541cdb303b0c02ce1b255cf23e2
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
ebb1846e064fca2e3496620c7eb6ed7d2f2310d8ce2329dd89971d3e4dcf50e6
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
2e09d88716795030b611ce3d0b3b4bd9d7b2d272c64456c6914c3bc98f167b1d

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer provider list
+---------+-------------------------------------------------+
| name    | description                                     |
+---------+-------------------------------------------------+
| amphora | The Octavia Amphora driver.                     |
| octavia | Deprecated alias of the Octavia Amphora driver. |
+---------+-------------------------------------------------+

# 2. OCP4.5 installed with amphora provider:


(overcloud) [stack@undercloud-0 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-06-24-055959   True        False         15m     Cluster version is 4.5.0-0.nightly-2020-06-24-055959
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list -f yaml
- id: c2feeaf2-de98-4642-ae03-74f15804e7d0
  name: ostest-vmhrz-kuryr-api-loadbalancer
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.0.1
- id: e6fa6469-d918-4132-be80-0538baa0c6e5
  name: openshift-cluster-version/cluster-version-operator
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.149.85
- id: 76b2d919-cd37-4b68-8716-d8abc2b295c9
  name: openshift-service-ca-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.94.151
- id: b7dce23e-e2e4-496e-b965-cd3d84e75b56
  name: openshift-kube-scheduler-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.173.121
- id: 969273b2-724c-43ff-981f-b2f9774cd189
  name: openshift-authentication-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.241.179
- id: 160c3c8a-1625-40eb-8550-261be639ebc4
  name: openshift-apiserver-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.213.32
- id: cef4f2bb-61e4-4031-8ff0-cddbab99b31b
  name: openshift-kube-controller-manager-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.94.127
- id: 1963cab9-223d-4e90-9a52-dc3b424baf12
  name: openshift-kube-apiserver-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.60.208
- id: d224be84-86de-465e-ace1-ec5d3d95f4ee
  name: openshift-etcd-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.207.110
- id: 6c421c3d-38f4-4d50-a475-9e1373db0298
  name: openshift-controller-manager-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.163.16
- id: 3f566489-31cd-469d-9ccc-f26faa2e111e
  name: openshift-kube-storage-version-migrator-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.161.28
- id: 9a1f5139-3f26-4b85-a2fd-1d14610d2009
  name: openshift-multus/multus-admission-controller
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.8.138
- id: 855b94fc-a27e-472e-ba2f-7bdba815a413
  name: openshift-dns-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.7.184
- id: bce476a8-a2d4-4657-82dd-c0be95afd7b1
  name: openshift-operator-lifecycle-manager/catalog-operator-metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.94.116
- id: 776a8758-e270-4b00-9edc-a8c9fdfe65b0
  name: openshift-etcd/etcd
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.144.138
- id: 08dc34d0-1e65-4837-8a44-fd388cfb8f71
  name: openshift-kube-scheduler/scheduler
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.49.216
- id: 0be11d5b-0fce-43c3-826f-93ba882d31fd
  name: openshift-dns/dns-default
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.0.10
- id: 6d03d6e2-0e8a-4709-9f21-739e339761bc
  name: openshift-kube-apiserver/apiserver
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.223.160
- id: e78e54af-d917-4e19-92f0-cc6e303f8ebd
  name: openshift-kube-controller-manager/kube-controller-manager
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.119.2
- id: a4ebda6d-2a48-4b9a-be91-225e20e6d8d5
  name: openshift-operator-lifecycle-manager/olm-operator-metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.5.84
- id: a202737b-4934-409b-a59a-e714ff1d31a7
  name: openshift-apiserver/api
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.213.18
- id: feabb23b-fbb1-4d31-92d8-c0ebccd5911a
  name: openshift-controller-manager/controller-manager
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.122.132
- id: 3c87c096-53de-468b-ae30-ba989b29aed9
  name: openshift-marketplace/marketplace-operator-metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.157.9
- id: 17c905ed-5bc1-467e-a057-deb75aa4f144
  name: openshift-cloud-credential-operator/cco-metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.250.19
- id: 69373b19-2218-49b2-9eca-bbcd3713ad42
  name: openshift-cloud-credential-operator/controller-manager-service
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.190.212
- id: e618bb9c-c8a5-445a-b7e0-27cdb6010afd
  name: openshift-insights/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.215.8
- id: 4a4f2c7e-42a4-4c3e-9674-022b03fdbf54
  name: openshift-ingress-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.63.119
- id: 7b538c7e-af40-4652-ba8a-715d71b44bea
  name: openshift-console-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.0.251
- id: 019a3aaf-3e8a-4b21-b86c-6afa48ca8bb8
  name: openshift-machine-api/machine-api-operator
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.152.180
- id: e6379813-1415-4dbe-9bb1-0fad8912337c
  name: openshift-console/downloads
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.186.132
- id: 803c0e5f-88c9-4d17-8a5c-3e1eb3a4dc5f
  name: openshift-machine-api/cluster-autoscaler-operator
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.66.201
- id: b10e0a20-3e3b-4a42-92fa-9886bc02b909
  name: openshift-config-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.74.0
- id: b57c5627-aa0d-4a19-b599-c02e050543ef
  name: openshift-machine-config-operator/machine-config-daemon
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.242.126
- id: 88aaf4bd-294d-43a1-ba0c-f75753a73761
  name: openshift-cluster-storage-operator/csi-snapshot-controller-operator-metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.4.43
- id: 74972953-a56a-4fb3-b029-86f696a925d3
  name: openshift-monitoring/prometheus-adapter
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.109.87
- id: ff7cc1e2-6d17-451d-8946-cad8b563fa8f
  name: openshift-ingress/router-internal-default
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.88.64
- id: e36705e6-cab2-46ec-a591-ab6d9b3037f0
  name: openshift-image-registry/image-registry
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.35.88
- id: 04197970-0c83-4892-839c-5fb0135e3c26
  name: openshift-monitoring/grafana
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.243.154
- id: d56bd15b-d676-40bb-82ba-35baaf841be4
  name: openshift-monitoring/alertmanager-main
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.124.208
- id: c8b1c5e3-3896-43a0-b30b-dbdd340f996a
  name: openshift-monitoring/thanos-querier
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.206.124
- id: 2cc95315-f3ec-4d96-8805-83fc3b5c9b37
  name: openshift-authentication/oauth-openshift
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.197.135
- id: e5526f7e-e15d-4b5d-9a00-0386414e6b67
  name: openshift-marketplace/redhat-operators
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.65.38
- id: d148a434-dbd7-47dc-9b0f-30bad81e6c7b
  name: openshift-monitoring/prometheus-k8s
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.73.236
- id: c74215e9-52ad-4c7a-82fc-5036d6c2c340
  name: openshift-marketplace/certified-operators
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.225.176
- id: 917e4278-88fc-497e-9eb9-afacb794fa90
  name: openshift-marketplace/community-operators
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.209.91
- id: fe80e4e5-8dcf-453d-a338-90b7651fdf38
  name: openshift-marketplace/redhat-marketplace
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.136.74
- id: 34e5b688-99b2-4c9a-8c91-d76768c1b013
  name: openshift-console/console
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.154.155
- id: 5aad1356-8c8c-4b85-9372-b62533a71719
  name: openshift-operator-lifecycle-manager/packageserver-service
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.143.77

# 3. Check environment:

oc new-project test
oc run --image kuryr/demo demo
oc run --image kuryr/demo demo-allowed-caller
oc run --image kuryr/demo demo-caller
oc expose pod/demo --port 80 --target-port 8080


(overcloud) [stack@undercloud-0 ~]$ oc get all
NAME                      READY   STATUS    RESTARTS   AGE
pod/demo                  1/1     Running   0          2m50s
pod/demo-allowed-caller   1/1     Running   0          2m50s
pod/demo-caller           1/1     Running   0          2m50s

NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/demo   ClusterIP   172.30.90.154   <none>        80/TCP    2m49s
(overcloud) [stack@undercloud-0 ~]$ oc rsh pod/demo-caller curl 172.30.90.154
demo: HELLO! I AM ALIVE!!!

# 4. Restore ovn-octavia provider on controllers:

Modify octavia.conf with enabled_provider_drivers=amphora: The Octavia Amphora driver.,octavia: Deprecated alias of the Octavia Amphora driver.,ovn: Octavia OVN driver.


[root@controller-0 ~]# vi /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf
[root@controller-0 ~]# docker restart octavia_worker && docker restart octavia_housekeeping && docker restart octavia_api && docker restart octavia_health_manager
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
7b466c70d77806a31b7a7dc9877fd9560a7a624d15bfd8d4fc4c5b70ad10c424
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
bce35cb4431d9dfa64a5e9a90ebdc6a68cf700ea2dabf8ddcb87d5a81804629d
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
4337252d0f34cc505dbb22a86b8651cc558fb6da4ff87d8d47e72de4371dba78
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
c7b64a7a7e147336e6b4a09a8ae45ab813f99f5e01a7cf48d1a1a244660ce2b6

[root@controller-1 ~]# vi /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf
[root@controller-1 ~]# docker restart octavia_worker && docker restart octavia_housekeeping && docker restart octavia_api && docker restart octavia_health_manager
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
3d2987c0aa65931958e88e7f405747559cc8bb3f695aba312506a14850c1dbe5
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
e68f833fd9351f330cefbeade00cfba66a8c332ab319f436fcb60d8a4c927a9a
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
015385bbd8efc2d7057e5e1e0370f847b01831d05033b7715dd521e5ef6b6cf2
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
94de52ab929c32236a1b64b22fb30090b481763c0cfd66317d7c27142d8f18d2

[root@controller-2 ~]# vi /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf
[root@controller-2 ~]# docker restart octavia_worker && docker restart octavia_housekeeping && docker restart octavia_api && docker restart octavia_health_manager
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
036fce7d254a7350c0795b6b4e5f6344e03ccb49da405213e452898d71748ab8
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
8f87a6ca3c0d84dea34002b5a67da61d443b6541cdb303b0c02ce1b255cf23e2
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
ebb1846e064fca2e3496620c7eb6ed7d2f2310d8ce2329dd89971d3e4dcf50e6
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
2e09d88716795030b611ce3d0b3b4bd9d7b2d272c64456c6914c3bc98f167b1d

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer provider list
+---------+-------------------------------------------------+
| name    | description                                     |
+---------+-------------------------------------------------+
| amphora | The Octavia Amphora driver.                     |
| octavia | Deprecated alias of the Octavia Amphora driver. |
| ovn     | Octavia OVN driver.                             |
+---------+-------------------------------------------------+

# 5. Execute upgrade instructions (https://github.com/openshift/openshift-docs/pull/22878/files.)

# 5.1 Check ovn-octavia driver is available:

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer provider list
+---------+-------------------------------------------------+
| name    | description                                     |
+---------+-------------------------------------------------+
| amphora | The Octavia Amphora driver.                     |
| octavia | Deprecated alias of the Octavia Amphora driver. |
| ovn     | Octavia OVN driver.                             |
+---------+-------------------------------------------------+

# 5.2 Edit the kuryr-config ConfigMap to trigger the recreation. You simply need to delete the annotation that states the default kuryr-octavia-provider is being used.

(undercloud) [stack@undercloud-0 ~]$ oc -n openshift-kuryr edit cm kuryr-config
configmap/kuryr-config edited

Before:

kind: ConfigMap
metadata:
  annotations:
    networkoperator.openshift.io/kuryr-octavia-provider: default
    networkoperator.openshift.io/kuryr-octavia-version: v2.13
  creationTimestamp: "2020-06-24T14:53:02Z"

After:

kind: ConfigMap
metadata:
  annotations:
    networkoperator.openshift.io/kuryr-octavia-version: v2.13
  creationTimestamp: "2020-06-24T14:53:02Z"



# 5.3 Wait until Cluster Network Operator reconcile loop is executed and detects the modification, triggering the recreation of the kuryr-controller and kuryr-cni pods

# 5.4 Check kuryr-config ConfigMap annotation has been readded but indicating the ovn driver is being used:

kind: ConfigMap
metadata:
  annotations:
    networkoperator.openshift.io/kuryr-octavia-provider: ovn
    networkoperator.openshift.io/kuryr-octavia-version: v2.13
  creationTimestamp: "2020-06-24T14:53:02Z"

# 5.5 Wait until all the loadbalancers have been recreated. Only one amphora loadbalancer must remain (the one created by Cluster Network Operator) and the rest should be of ovn type:

overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list -f yaml
- id: c2feeaf2-de98-4642-ae03-74f15804e7d0
  name: ostest-vmhrz-kuryr-api-loadbalancer
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 172.30.0.1
- id: 89286d01-1061-4f09-9a16-8d1c2eb640df
  name: openshift-cluster-version/cluster-version-operator
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.149.85
- id: a1226172-6866-47a7-a139-a5dc8f645163
  name: openshift-apiserver-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.213.32
- id: 85683d25-0666-49cf-a4e1-a1cd27edb26c
  name: openshift-authentication-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.241.179
- id: 78b5ee05-13f8-4e77-82d8-7582da8999a1
  name: openshift-etcd-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.207.110
- id: 456966a7-6497-49f4-9657-d1c7199eb4f1
  name: openshift-kube-scheduler-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.173.121
- id: 243b10af-577d-405b-bdf6-d971425dbe17
  name: openshift-controller-manager-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.163.16
- id: 0233b921-a323-47d2-bf27-20b1e973e953
  name: openshift-operator-lifecycle-manager/catalog-operator-metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.94.116
- id: 5209d2de-7a69-4f44-993a-b23116b49e9f
  name: openshift-kube-controller-manager-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.94.127
- id: a8f5843d-a6bf-422c-aa96-91680b33f552
  name: openshift-service-ca-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.94.151
- id: 0efc0047-97f1-4f67-b174-05c843be529e
  name: openshift-multus/multus-admission-controller
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.8.138
- id: 6bca9e3a-837e-4bd7-9a21-82db91aedeae
  name: openshift-kube-apiserver-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.60.208
- id: bcc39809-55e7-46db-85b7-2a5c20b78fec
  name: openshift-dns/dns-default
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.0.10
- id: d78a7403-3779-46b7-8a07-22232daaccb7
  name: openshift-kube-scheduler/scheduler
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.49.216
- id: 1f34db67-239e-4b8f-9864-39c33bddb85d
  name: openshift-kube-storage-version-migrator-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.161.28
- id: 5b648255-7cc4-47fc-92bf-0763c6411278
  name: openshift-kube-apiserver/apiserver
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.223.160
- id: 4ff77b8d-16b7-4858-af60-1fe167492fe6
  name: openshift-marketplace/marketplace-operator-metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.157.9
- id: b1a4ef48-c306-4bdc-b3db-983cd21b16b7
  name: openshift-dns-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.7.184
- id: 1519aa37-28dd-4f2d-8bc1-c8a76f747455
  name: openshift-etcd/etcd
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.144.138
- id: e883db09-3aef-4ac3-80d0-ed6ceddd72b2
  name: openshift-apiserver/api
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.213.18
- id: ffe7af5d-e234-4490-b452-a8da4dad76d6
  name: openshift-kube-controller-manager/kube-controller-manager
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.119.2
- id: ade21d90-0e32-4c62-854a-164e5dc0c6ae
  name: openshift-operator-lifecycle-manager/olm-operator-metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.5.84
- id: 959782ca-45f2-4483-8876-da49e3dd2571
  name: openshift-controller-manager/controller-manager
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.122.132
- id: 2d1cedf2-78cc-4943-92d0-a818c5532514
  name: openshift-cloud-credential-operator/cco-metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.250.19
- id: e2337b53-1a76-48d9-b930-9cc2b36b9cab
  name: openshift-ingress-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.63.119
- id: 70626880-1db1-4638-a50f-30456b20c974
  name: openshift-console/downloads
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.186.132
- id: 97469cf8-2d3f-4764-8eae-f9406fb6746e
  name: openshift-config-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.74.0
- id: e617b1db-92db-42b7-a839-49455f842e8b
  name: openshift-insights/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.215.8
- id: 2083e58d-8ab9-4cd2-aaa5-54394732b1d6
  name: openshift-machine-api/machine-api-operator
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.152.180
- id: e95410b0-1f07-4800-8f4f-7a2be6d32f03
  name: openshift-machine-api/cluster-autoscaler-operator
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.66.201
- id: 192e8f0c-8aba-4399-b007-7fe89da492fd
  name: openshift-marketplace/redhat-operators
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.65.38
- id: 67c2a7b6-8493-4e0e-b70f-a89c9592d8b7
  name: openshift-console/console
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.154.155
- id: 22b63fcd-30f9-46a8-ab37-74810c3680f1
  name: openshift-machine-config-operator/machine-config-daemon
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.242.126
- id: e88c744f-5be5-4523-a7a0-db4e8e4a0d06
  name: openshift-marketplace/certified-operators
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.225.176
- id: 7740e110-cf70-4a03-add4-e8eea28b6363
  name: openshift-monitoring/grafana
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.243.154
- id: 931bd2d0-38d4-460a-9c73-73dc70e1c594
  name: openshift-ingress/router-internal-default
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.88.64
- id: 84591e49-200b-4f3b-92a5-63f45c2f708e
  name: openshift-image-registry/image-registry
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.35.88
- id: 2376e115-9e73-4e73-9bed-fc4dfd88a321
  name: openshift-monitoring/alertmanager-main
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.124.208
- id: 95343516-1fd6-42aa-8d5a-cdf2afc66282
  name: openshift-monitoring/prometheus-adapter
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.109.87
- id: b84fc205-25c3-4c92-8740-0dddd9d22ca7
  name: openshift-operator-lifecycle-manager/packageserver-service
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.143.77
- id: b8ec7bc2-c57f-4223-b58c-e3722ea1478f
  name: openshift-marketplace/redhat-marketplace
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.136.74
- id: a2c5c027-350f-421b-bb16-17e136cfe6b0
  name: openshift-authentication/oauth-openshift
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.197.135
- id: 1893f9e6-09f1-476f-89f1-50fbcc250422
  name: openshift-monitoring/prometheus-k8s
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.73.236
- id: f07b98d8-76f2-463b-924c-6001430d9cde
  name: openshift-cloud-credential-operator/controller-manager-service
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.190.212
- id: 42129ad8-e87d-44ff-a7b6-5727bae92672
  name: openshift-marketplace/community-operators
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.209.91
- id: 79737c41-4cee-471b-a152-d6da126e9149
  name: openshift-console-operator/metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.0.251
- id: f55801f2-ba7d-4aba-80ff-58218fa9498e
  name: openshift-monitoring/thanos-querier
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.206.124
- id: 0dc8ae37-0009-4bba-9dbb-cdedd35916bc
  name: openshift-cluster-storage-operator/csi-snapshot-controller-operator-metrics
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.4.43
- id: d8bf3c32-e109-489e-8229-568256c72a10
  name: test/demo
  project_id: 8c46f20cb5904470bc031b0e456886f0
  provider: ovn
  provisioning_status: ACTIVE
  vip_address: 172.30.90.154


# 6. Check that service previously created is still operative with ovn provider:

(overcloud) [stack@undercloud-0 ~]$ oc rsh pod/demo-caller curl 172.30.90.154
demo: HELLO! I AM ALIVE!!!

----
Kuryr controller logs during upgrade attached.
Comment 6 errata-xmlrpc 2020-07-13 17:44:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409