Bug 2035326 - [ExternalCloudProvider] installation with additional network on workers fails
Summary: [ExternalCloudProvider] installation with additional network on workers fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.10
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 4.10.0
Assignee: Andreas Karis
QA Contact: rlobillo
URL:
Whiteboard:
: 2036112 (view as bug list)
Depends On:
Blocks: 2044503
TreeView+ depends on / blocked
 
Reported: 2021-12-23 15:49 UTC by rlobillo
Modified: 2022-03-10 16:36 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2044503 (view as bug list)
Environment:
Last Closed: 2022-03-10 16:36:23 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
must-gather (12.22 MB, application/gzip)
2021-12-23 15:49 UTC, rlobillo
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2898 0 None open Bug 2035326: ovs-configuration - use lower than NM default ethernet route metric 2021-12-28 19:25:15 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:36:34 UTC

Description rlobillo 2021-12-23 15:49:43 UTC
Created attachment 1847602 [details]
must-gather

Description of problem:

The OCP installation on top of Openstack is failing as the workers are no attached to the cluster. 

The workers have an extra network that is attached to all of them (for integration with Manila) and it is provoking a wrong routing table on the workers, so they are not able to connect to internet an download the images to get configured.

The install-config.yaml looks like below:

$ cat install-config.yaml                                                                                                                                 
# This file is autogenerated by infrared openshift plugin                                                                                                                                    
apiVersion: v1                                                                                                                                                                               
baseDomain: "shiftstack.com"                                                                                                                                                                 
compute:                                                                                                                                                                                     
- name: worker                                                                                                                                                                               
  platform:                                                                                                                                                                                  
    openstack:                                                                                                                                                                               
      zones: []                                                                                                                                                                              
      additionalNetworkIDs: ['c1185e5e-ac6d-4548-bafc-d7326b261c5c']                                                                                                                         
  replicas: 3                                                                                                                                                                                
controlPlane:                                                                                                                                                                                
  name: master
  platform:
    openstack:                                                                                                                                                                               
      zones: []                                                                                                                                                                              
  replicas: 3                                                                                                                                                                                
metadata:
  name: "ostest"
networking:
  clusterNetworks:
  - cidr:             10.128.0.0/14
    hostSubnetLength: 9
  serviceNetwork:
    - 172.30.0.0/16
  machineNetwork:
    - cidr: "10.196.0.0/16"
  networkType: "OVNKubernetes"
[...]

where:

$ openstack subnet list | grep  c1185e5e-ac6d-4548-bafc-d7326b261c5c
| 3d24e0d0-05ca-4659-ada7-86f83b2eceb1 | StorageNFSSubnet   | c1185e5e-ac6d-4548-bafc-d7326b261c5c | 172.17.5.0/24 |

Before installation we are manipulating the manifest to enable the externalCCM on it:

      apiVersion: config.openshift.io/v1
      kind: FeatureGate
      metadata:
        annotations:
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          release.openshift.io/create-only: "true"
        name: cluster
      spec:
        customNoUpgrade:
          enabled:
          - ExternalCloudProvider
          - CSIMigrationOpenStack
        featureSet: CustomNoUpgrade

The installation is failing due to the machines remaining on provisioned status, and the node resource is not created:

(shiftstack) [stack@undercloud-0 ~]$ oc get machines -n openshift-machine-api
NAME                          PHASE         TYPE        REGION      ZONE   AGE
ostest-td7wn-master-0         Running                                      19h
ostest-td7wn-master-1         Running                                      19h
ostest-td7wn-master-2         Running                                      19h
ostest-td7wn-worker-0-64prw   Provisioned                                  19h
ostest-td7wn-worker-0-78pvd   Provisioned   m4.xlarge   regionOne   nova   4h33m
ostest-td7wn-worker-0-rxccv   Provisioned   m4.xlarge   regionOne   nova   19h
(shiftstack) [stack@undercloud-0 ~]$ oc get nodes
NAME                    STATUS   ROLES    AGE   VERSION
ostest-td7wn-master-0   Ready    master   19h   v1.22.1+6859754
ostest-td7wn-master-1   Ready    master   19h   v1.22.1+6859754
ostest-td7wn-master-2   Ready    master   19h   v1.22.1+6859754

(shiftstack) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+-----------------------------+--------+--------------------------------------------------------------+--------------------+--------+
| ID                                   | Name                        | Status | Networks                                                     | Image              | Flavor |
+--------------------------------------+-----------------------------+--------+--------------------------------------------------------------+--------------------+--------+
| 0c792e79-481e-4653-953f-d3b4e451834c | ostest-td7wn-worker-0-78pvd | ACTIVE | StorageNFS=172.17.5.164; ostest-td7wn-openshift=10.196.2.210 | ostest-td7wn-rhcos |        |
| 835d86d4-16ce-4d72-82cd-658aa61ff354 | ostest-td7wn-worker-0-rxccv | ACTIVE | StorageNFS=172.17.5.209; ostest-td7wn-openshift=10.196.2.187 | ostest-td7wn-rhcos |        |
| 16bf0272-aa74-4c0c-8769-990756163201 | ostest-td7wn-worker-0-64prw | ACTIVE | StorageNFS=172.17.5.193; ostest-td7wn-openshift=10.196.2.119 | ostest-td7wn-rhcos |        |
| c7f64b3a-48a9-489c-bab7-1e1356e149fd | ostest-td7wn-master-2       | ACTIVE | ostest-td7wn-openshift=10.196.1.25                           | ostest-td7wn-rhcos |        |
| f1e579f3-8457-47c3-9b7b-8fa302c8e2d5 | ostest-td7wn-master-1       | ACTIVE | ostest-td7wn-openshift=10.196.2.93                           | ostest-td7wn-rhcos |        |
| 8415da41-4c86-4d48-ac5f-6def6b30bea2 | ostest-td7wn-master-0       | ACTIVE | ostest-td7wn-openshift=10.196.0.150                          | ostest-td7wn-rhcos |        |
+--------------------------------------+-----------------------------+--------+--------------------------------------------------------------+--------------------+--------+

The workers have no access to internet due to a routing issue:

(shiftstack) [stack@undercloud-0 ~]$ ssh -J core.43.181 core.2.187
Warning: Permanently added '10.46.43.181' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.196.2.187' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 410.84.202112201602-0
  Part of OpenShift 4.10, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.10/architecture/architecture-rhcos.html

---
[core@ostest-td7wn-worker-0-rxccv ~]$ sudo netstat -nrv
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.17.5.1      0.0.0.0         UG        0 0          0 ens4
0.0.0.0         10.196.0.1      0.0.0.0         UG        0 0          0 br-ex
10.196.0.0      0.0.0.0         255.255.0.0     U         0 0          0 br-ex
169.254.169.254 172.17.5.150    255.255.255.255 UGH       0 0          0 ens4
169.254.169.254 10.196.0.10     255.255.255.255 UGH       0 0          0 br-ex
172.17.5.0      0.0.0.0         255.255.255.0   U         0 0          0 ens4
[core@ostest-td7wn-worker-0-rxccv ~]$ ip r
default via 172.17.5.1 dev ens4 proto dhcp metric 100 
default via 10.196.0.1 dev br-ex proto dhcp metric 100 
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.2.187 metric 100 
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 100 
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.209 metric 100 

[core@ostest-td7wn-worker-0-rxccv ~]$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1032ms

The first entry on the routing table is sending the default traffic to the manila network, instead of the machine network.

Removing the ens4 default route fix the issue:

[core@ostest-td7wn-worker-0-rxccv ~]$ sudo ip route delete default dev ens4
[core@ostest-td7wn-worker-0-rxccv ~]$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=114 time=41.5 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=114 time=40.3 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=114 time=39.8 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 39.785/40.544/41.536/0.733 ms

The same type of installation is working using:
1. the in-tree Cloud Config Manager with OVNKubernetes 
2. ExternalCloudProvider and OpenShiftSDN NetworkType.

In the second case, the routes on the worker looks like this:

[core@ostest-46nr7-worker-0-xmlcw ~]$ ip r
default via 10.196.0.1 dev ens3 proto dhcp metric 100 
default via 172.17.5.1 dev ens4 proto dhcp metric 101 
10.128.0.0/14 dev tun0 scope link 
10.196.0.0/16 dev ens3 proto kernel scope link src 10.196.0.65 metric 100 
169.254.169.254 via 10.196.0.10 dev ens3 proto dhcp metric 100 
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 101 
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.164 metric 101 
172.30.0.0/16 dev tun0 

where the defaults are created with different metrics.





Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2021-12-21-130047 on top of RHOS-16.2-RHEL-8-20211129.n.1 with Manila and OVN enabled.


How reproducible: Always.


Steps to Reproduce:
1. Install as detailed on description.
2.
3.

Actual results: Installation fails.


Expected results: Installation OK.


Additional info: 
- must-gather.
- install-config.yaml

Comment 2 Andreas Karis 2021-12-28 11:02:57 UTC
Hi there,

Can you provide the `openstack network show`, `openstack subnet show`, `openstack router show` command for the networks, subnets, routers in question? Please also add the `openstack port show` for all worker and master node ports, as well as the `openstack server show` for all master and worker nodes.

How are you telling OpenStack / OpenShift which of the 2 default routes should have a better metric than the other?

Which version of OpenStack are you using underneath OpenShift?

Thanks!

- Andreas

Comment 3 Andreas Karis 2021-12-28 18:17:49 UTC
I managed to grab an OpenStack 16.2 system (all virtual, so it's not powerful enough for an OpenShift deployment) and I created an instance with (incomplete reproducer steps):
~~~
openstack network create akaris1
openstack network create akaris2
openstack subnet create akaris1-subnet --network akaris1 --subnet-range 192.168.20.0/24 --dns-nameserver 10.5.30.45
openstack subnet create akaris2-subnet --network akaris2 --subnet-range 192.168.21.0/24 --dns-nameserver 10.5.30.45
openstack router add subnet test-router-andrea akaris1-subnet
openstack router add subnet test-router-andrea akaris2-subnet
openstack server create --image rhel8 --flavor nano --nic net-id=akaris1 --nic net-id=akaris2 rhel8
openstack floating ip set --port 9891d857-6c6e-4e05-a090-c5d58d291c7a 10.0.0.172
~~~

It turns out that is actually a NetworkManager feature and neither one from OpenStack nor form OpenShift or OVN-Kubernetes. Nothing in the interface configuration or the DHCP leases which are handed out from dnsmasq actually sets a metric. Instead:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/managing-the-default-gateway-setting_configuring-and-managing-networking 
~~~
21.8. How NetworkManager manages multiple default gateways

In certain situations, for example for fallback reasons, you set multiple default gateways on a host. However, to avoid asynchronous routing issues, each default gateway of the same protocol requires a separate metric value. Note that RHEL only uses the connection to the default gateway that has the lowest metric set.

You can set the metric for both the IPv4 and IPv6 gateway of a connection using the following command:

# nmcli connection modify connection-name ipv4.route-metric value ipv6.route-metric value

Important

Do not set the same metric value for the same protocol in multiple connection profiles to avoid routing issues.

If you set a default gateway without a metric value, NetworkManager automatically sets the metric value based on the interface type. For that, NetworkManager assigns the default value of this network type to the first connection that is activated, and sets an incremented value to each other connection of the same type in the order they are activated. For example, if two Ethernet connections with a default gateway exist, NetworkManager sets a metric of 100 on the route to the default gateway of the connection that you activate first. For the second connection, NetworkManager sets 101. 
~~~

For OpenShiftSDN, this works because both interfaces are of the same type Ethernet. The first attached network is the machine network so it gets 100, and the second attached network is the manila network, so it gets a route metric of 101.
However, there's really no guarantee that this works. It's just thanks to order of operations that the correct metric is assigned to the respective interfaces.

For OVNKubernetes this fails because br-ex and ens4 are of 2 different types.

The conversion happens here https://github.com/openshift/machine-config-operator/blob/master/templates/common/_base/files/configure-ovs-network.yaml#L511
We force a metric of 100 for br-ex here in the conversion process:
https://github.com/openshift/machine-config-operator/blob/master/templates/common/_base/files/configure-ovs-network.yaml#L183

That's the same as the default ethernet route metric. Given that ens4 is now the only ethernet type interface, it's metric is lowered from 101 to 100, and we suddently have 2 routes with the same metric of value 100.

To test this further, I installed openvswitch2.16, NetworkManager-ovs and I converted the interfaces on my RHEL instance with the same script that's used by OpenShift:
https://github.com/openshift/machine-config-operator/blob/master/templates/common/_base/files/configure-ovs-network.yaml

In order to reset the interfaces after each test, I used the following procedure:
~~~
# delete all connections
nmcli conn delete <...>
cloud-init clean
cloud-init init
# if needed copy ifcfg-eth0 script to ifcfg-eth1
systemctl restart NetworkManager
~~~

~~~
[root@rhel8 ~]# ip r
default via 192.168.20.1 dev eth0 proto dhcp metric 100 
default via 192.168.21.1 dev eth1 proto dhcp metric 101 
169.254.169.254 via 192.168.20.2 dev eth0 proto dhcp metric 100 
169.254.169.254 via 192.168.21.2 dev eth1 proto dhcp metric 101 
192.168.20.0/24 dev eth0 proto kernel scope link src 192.168.20.80 metric 100 
192.168.21.0/24 dev eth1 proto kernel scope link src 192.168.21.167 metric 101 
~~~

~~~
/usr/local/bin/ovs-configure.sh
~~~

Right after executing the script:
~~~
# ip r
default via 192.168.21.1 dev eth1 proto dhcp metric 100 
default via 192.168.20.1 dev br-ex proto dhcp metric 100 
169.254.169.254 via 192.168.21.2 dev eth1 proto dhcp metric 100 
169.254.169.254 via 192.168.20.2 dev br-ex proto dhcp metric 100 
192.168.20.0/24 dev br-ex proto kernel scope link src 192.168.20.80 metric 100 
192.168.21.0/24 dev eth1 proto kernel scope link src 192.168.21.167 metric 100 
~~~

~~~
# nmcli conn show br-ex
(...)
connection.type:                        ovs-bridge
(...)
~~~

~~~
# nmcli conn show 'System eth1'
(...)
connection.type:                        802-3-ethernet
(...)
~~~

A very similar issue was already addressed here:
https://bugzilla.redhat.com/show_bug.cgi?id=1880259

Fixes to that BZ which set the metric to 100:
https://github.com/openshift/machine-config-operator/commit/974bb6fae2cb3bc0beff8f43e3879cf22d95c148
https://github.com/openshift/machine-config-operator/commit/d6326cd110bcfacfb52912f4696335ccefc193b7

Without the above fixes, the routing table had a metric of 800:
~~~
# ip r
default via 192.168.21.1 dev eth1 proto dhcp metric 100 
default via 192.168.20.1 dev br-ex proto dhcp metric 800 
169.254.169.254 via 192.168.21.2 dev eth1 proto dhcp metric 100 
169.254.169.254 via 192.168.20.2 dev br-ex proto dhcp metric 800 
192.168.20.0/24 dev br-ex proto kernel scope link src 192.168.20.80 metric 800 
192.168.21.0/24 dev eth1 proto kernel scope link src 192.168.21.167 metric 100
~~~

Simply lowering the metric to e.g. 50 should be the easiest fix for this and would be in line with the intention behind the above fixes.

Comment 8 Stephen Finucane 2022-01-20 15:25:31 UTC
*** Bug 2036112 has been marked as a duplicate of this bug. ***

Comment 11 zhaozhanqi 2022-01-25 03:31:38 UTC
@rlobillo Could you help verify this bug ,thanks.

Comment 13 rlobillo 2022-01-25 09:35:00 UTC
Verified on OCP4.10.0-0.nightly-2022-01-25-023600 on top of OSP16.1 (RHOS-16.1-RHEL-8-20210903.n.0)

Regular IPI installation with OVNKubernetes Network types finished successfully:

$ tail ~/ostest/.openshift_install.log 
time="2022-01-25T08:14:26Z" level=info msg="Access the OpenShift web-console here: https://console-openshift-console.apps.ostest.shiftstack.com"
time="2022-01-25T08:14:26Z" level=info msg="Login to the console with user: \"kubeadmin\", and password: \"TUMeD-NnoG2-547Nr-MZrp7\""
time="2022-01-25T08:14:26Z" level=debug msg="Time elapsed per stage:"
time="2022-01-25T08:14:26Z" level=debug msg="           masters: 2m17s"
time="2022-01-25T08:14:26Z" level=debug msg="         bootstrap: 44s"
time="2022-01-25T08:14:26Z" level=debug msg="Bootstrap Complete: 16m0s"
time="2022-01-25T08:14:26Z" level=debug msg="               API: 2m50s"
time="2022-01-25T08:14:26Z" level=debug msg=" Bootstrap Destroy: 34s"
time="2022-01-25T08:14:26Z" level=debug msg=" Cluster Operators: 21m16s"
time="2022-01-25T08:14:26Z" level=info msg="Time elapsed: 47m13s"

$ oc get co
NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.0-0.nightly-2022-01-25-023600   True        False         False      76m     
baremetal                                  4.10.0-0.nightly-2022-01-25-023600   True        False         False      100m    
cloud-controller-manager                   4.10.0-0.nightly-2022-01-25-023600   True        False         False      105m    
cloud-credential                           4.10.0-0.nightly-2022-01-25-023600   True        False         False      107m    
cluster-autoscaler                         4.10.0-0.nightly-2022-01-25-023600   True        False         False      100m    
config-operator                            4.10.0-0.nightly-2022-01-25-023600   True        False         False      101m    
console                                    4.10.0-0.nightly-2022-01-25-023600   True        False         False      75m     
csi-snapshot-controller                    4.10.0-0.nightly-2022-01-25-023600   True        False         False      100m    
dns                                        4.10.0-0.nightly-2022-01-25-023600   True        False         False      100m    
etcd                                       4.10.0-0.nightly-2022-01-25-023600   True        False         False      99m     
image-registry                             4.10.0-0.nightly-2022-01-25-023600   True        False         False      82m     
ingress                                    4.10.0-0.nightly-2022-01-25-023600   True        False         False      82m     
insights                                   4.10.0-0.nightly-2022-01-25-023600   True        False         False      94m     
kube-apiserver                             4.10.0-0.nightly-2022-01-25-023600   True        False         False      80m     
kube-controller-manager                    4.10.0-0.nightly-2022-01-25-023600   True        False         False      98m     
kube-scheduler                             4.10.0-0.nightly-2022-01-25-023600   True        False         False      98m     
kube-storage-version-migrator              4.10.0-0.nightly-2022-01-25-023600   True        False         False      101m    
machine-api                                4.10.0-0.nightly-2022-01-25-023600   True        False         False      96m     
machine-approver                           4.10.0-0.nightly-2022-01-25-023600   True        False         False      100m    
machine-config                             4.10.0-0.nightly-2022-01-25-023600   True        False         False      99m     
marketplace                                4.10.0-0.nightly-2022-01-25-023600   True        False         False      100m    
monitoring                                 4.10.0-0.nightly-2022-01-25-023600   True        False         False      75m     
network                                    4.10.0-0.nightly-2022-01-25-023600   True        False         False      102m    
node-tuning                                4.10.0-0.nightly-2022-01-25-023600   True        False         False      100m    
openshift-apiserver                        4.10.0-0.nightly-2022-01-25-023600   True        False         False      83m     
openshift-controller-manager               4.10.0-0.nightly-2022-01-25-023600   True        False         False      91m     
openshift-samples                          4.10.0-0.nightly-2022-01-25-023600   True        False         False      78m     
operator-lifecycle-manager                 4.10.0-0.nightly-2022-01-25-023600   True        False         False      100m    
operator-lifecycle-manager-catalog         4.10.0-0.nightly-2022-01-25-023600   True        False         False      100m    
operator-lifecycle-manager-packageserver   4.10.0-0.nightly-2022-01-25-023600   True        False         False      96m     
service-ca                                 4.10.0-0.nightly-2022-01-25-023600   True        False         False      101m    
storage                                    4.10.0-0.nightly-2022-01-25-023600   True        False         False      82m     

$ oc get nodes 
NAME                          STATUS   ROLES    AGE    VERSION
ostest-48ng8-master-0         Ready    master   106m   v1.23.0+06791f6
ostest-48ng8-master-1         Ready    master   106m   v1.23.0+06791f6
ostest-48ng8-master-2         Ready    master   106m   v1.23.0+06791f6
ostest-48ng8-worker-0-gqwjt   Ready    worker   85m    v1.23.0+06791f6
ostest-48ng8-worker-0-qblvj   Ready    worker   77m    v1.23.0+06791f6
ostest-48ng8-worker-0-zxjvg   Ready    worker   77m    v1.23.0+06791f6

$ oc get pods -n openshift-ovn-kubernetes
NAME                   READY   STATUS    RESTARTS       AGE
ovnkube-master-b6dld   6/6     Running   7 (91m ago)    105m
ovnkube-master-f7cdx   6/6     Running   2 (93m ago)    105m
ovnkube-master-jxln2   6/6     Running   6 (104m ago)   105m
ovnkube-node-56qs6     5/5     Running   0              77m
ovnkube-node-9wv6v     5/5     Running   3 (84m ago)    85m
ovnkube-node-9x7pq     5/5     Running   0              105m
ovnkube-node-h66j5     5/5     Running   0              105m
ovnkube-node-j82zs     5/5     Running   0              105m
ovnkube-node-llmsz     5/5     Running   0              78m

and the workers show the proper metrics:

[core@ostest-48ng8-worker-0-qblvj ~]$ ip r
default via 10.196.0.1 dev br-ex proto dhcp metric 49 
default via 172.17.5.1 dev ens4 proto dhcp metric 100 
10.128.0.0/14 via 10.128.2.1 dev ovn-k8s-mp0 
10.128.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.128.2.2 
10.196.0.0/16 dev br-ex proto kernel scope link src 10.196.0.156 metric 49 
169.254.169.0/30 via 10.196.0.1 dev br-ex 
169.254.169.3 via 10.128.2.1 dev ovn-k8s-mp0 
169.254.169.254 via 10.196.0.10 dev br-ex proto dhcp metric 49 
169.254.169.254 via 172.17.5.150 dev ens4 proto dhcp metric 100 
172.17.5.0/24 dev ens4 proto kernel scope link src 172.17.5.245 metric 100 
172.30.0.0/16 via 10.196.0.1 dev br-ex mtu 1342 

Furthermore, manila integration is working as expected:

$ oc get pods -n manila-test -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE                          NOMINATED NODE   READINESS GATES
demo-59c59d6d55-ct28v   1/1     Running   0          31s   10.129.2.18   ostest-48ng8-worker-0-zxjvg   <none>           <none>
demo-59c59d6d55-wpm8j   1/1     Running   0          31s   10.128.2.13   ostest-48ng8-worker-0-qblvj   <none>           <none>

$ oc get pvc -n manila-test -o wide
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE   VOLUMEMODE
pvc-manila   Bound    pvc-7b698c68-beb3-446d-af4a-70b030e3500c   1Gi        RWO            csi-manila-default   40s   Filesystem
(venv_shade) (shiftstack) [stack@undercloud-0 ~]$

Comment 16 errata-xmlrpc 2022-03-10 16:36:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.