Bug 2030733

This bug feels like a can of worms. Firstly I don't actually know how oc debug works, but to make my assumptions explicit, I am guessing it:

* Starts a pod on a node with host networking
* Uses the PodIP assigned to that pod by kubelet
* Connects to the PodIP internally from... presumably an apiserver.

The docs [1] describe InternalIP as "Typically the IP address of the node that is routable only within the cluster", but also "The usage of these fields varies depending on your cloud provider or bare metal configuration.". This seems a bit vague, but my interpretation is that it lists endpoint addresses for this node for internal communication, and a storage network exposed on a subset of nodes probably doesn't meet that criterion.

The comment on GetNodeHostIPs() in kubernetes/kubernetes [2] suggests it's the immediate source of oc debug's errant IP.

We should separate the 2 different lists of Addresses here:

The Machine object has a list of Addresses. These are not directly used by kubernetes, but define a list of IP addresses which will be approved if requested in a CSR generated by the kubelet running on the host. This list is generated by CAPO.

The Node object has a list of Addresses. These are the actual addresses used by kubernetes. These must be a (non-strict) subset of the Addresses defined on the Machine object, or kubelet will fail to come up when its CSR is not approved. These addresses are written by either kubelet (legacy cloud provider) or the CCM (external cloud provider).

This bug concerns the list of addresses on the Node object, and is therefore a cloud provider issue, not a CAPO issue.

For investigation:

* What's the implementation difference between legacy cloud provider (OpenStack) and CCM (OpenStack)
* What metadata is available to CCM to distinguish 'cluster' network(s) from infra networks?

[1] https://kubernetes.io/docs/concepts/architecture/nodes/#addresses
[2] https://github.com/kubernetes/kubernetes/blob/cc6f12583f2b611e9469a6b2e0247f028aae246b/pkg/util/node/node.go#L89-L93

Setting the priority as "medium". This must be properly investigated (and possibly resolved) before GA, which is not imminent.

NodeAddresses are generated quite differently on legacy vs external cloud provider.

Legacy cloud provider:
https://github.com/kubernetes/legacy-cloud-providers/blob/1a100831c5a0718b3ef6ae88bf506d383d387b45/openstack/openstack.go#L565-L626


External cloud provider:
https://github.com/kubernetes/cloud-provider-openstack/blob/d46aa87534042ad1e26b812d1ef1aa140317a25e/pkg/openstack/instances.go#L458-L565

where interfaces is provided by:
https://github.com/kubernetes/cloud-provider-openstack/blob/d46aa87534042ad1e26b812d1ef1aa140317a25e/pkg/openstack/instances.go#L611-L629

From git spelunking, this seems relevant: https://github.com/kubernetes/cloud-provider-openstack/issues/407

Ok, we need to fix this by setting internal-network-name in cloud.conf.

We currently don't have a mechanism to customise cloud.conf, so it's not yet possible to fix. However, we need to allow this anyway as a matter of urgency before we can GA this feature. Once we have the ability to customise cloud.conf this should be a relatively simple fix.

Removing the Triaged keyword because:
* the QE automation assessment (flag qe_test_coverage) is missing

*** Bug 2045493 has been marked as a duplicate of this bug. ***

*** Bug 2043659 has been marked as a duplicate of this bug. ***

Verified on 4.11.0-0.nightly-2022-04-08-205307 on top of RHOS-16.2-RHEL-8-20220311.n.1 installing OCP cluster with eCCM enabled on featureGate:

$ oc get featureGate/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
  annotations:
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    release.openshift.io/create-only: "true"
  creationTimestamp: "2022-04-11T13:29:12Z"
  generation: 1
  name: cluster
  resourceVersion: "1379"
  uid: 5766ece9-3dcc-4982-b355-f4c37d739ab9
spec:
  customNoUpgrade:
    enabled:
    - ExternalCloudProvider
  featureSet: CustomNoUpgrade

$ oc get pods -n openshift-cloud-controller-manager
NAME                                                  READY   STATUS    RESTARTS   AGE
openstack-cloud-controller-manager-7f7f67c5f8-lb4wx   1/1     Running   0          65m
openstack-cloud-controller-manager-7f7f67c5f8-sh9h9   1/1     Running   0          65m

---------------------------

Once the cluster is deployed through IPI, all the machines includes the primarySubnet:

$ oc get machine -n openshift-machine-api -o json | jq .items[].spec.providerSpec.value.primarySubnet
"f94ecb70-604a-447f-896b-6fc40b045e4c"
"f94ecb70-604a-447f-896b-6fc40b045e4c"
"f94ecb70-604a-447f-896b-6fc40b045e4c"
"f94ecb70-604a-447f-896b-6fc40b045e4c"
"f94ecb70-604a-447f-896b-6fc40b045e4c"
"f94ecb70-604a-447f-896b-6fc40b045e4c"

And the workers have two networks defined:

$ oc get machines -n openshift-machine-api -l machine.openshift.io/cluster-api-machine-role=worker -o json | jq .items[].spec.providerSpec.value.networks
[
  {
    "filter": {},
    "subnets": [
      {
        "filter": {},
        "uuid": "f94ecb70-604a-447f-896b-6fc40b045e4c"
      }
    ]
  },
  {
    "filter": {},
    "noAllowedAddressPairs": true,
    "uuid": "b37bbd3d-e5f9-45ce-a9f9-6934f8f7d0ac"
  }
]
[
  {
    "filter": {},
    "subnets": [
      {
        "filter": {},
        "uuid": "f94ecb70-604a-447f-896b-6fc40b045e4c"
      }
    ]
  },
  {
    "filter": {},
    "noAllowedAddressPairs": true,
    "uuid": "b37bbd3d-e5f9-45ce-a9f9-6934f8f7d0ac"
  }
]
[
  {
    "filter": {},
    "subnets": [
      {
        "filter": {},
        "uuid": "f94ecb70-604a-447f-896b-6fc40b045e4c"
      }
    ]
  },
  {
    "filter": {},
    "noAllowedAddressPairs": true,
    "uuid": "b37bbd3d-e5f9-45ce-a9f9-6934f8f7d0ac"
  }
]

where the primarySubnet is:

$ openstack subnet list | grep f94ecb70-604a-447f-896b-6fc40b045e4c
| f94ecb70-604a-447f-896b-6fc40b045e4c | restricted_subnet     | 059e58b8-fd1c-41d1-b44c-d7fced04d078 | 172.16.0.0/24  |

and the secondary network is the one used for integrating with Manila:

$ openstack subnet list | grep b37bbd3d-e5f9-45ce-a9f9-6934f8f7d0ac
| 5dbe57da-73ea-457f-b044-8f05459d9368 | StorageNFSSubnet      | b37bbd3d-e5f9-45ce-a9f9-6934f8f7d0ac | 172.17.5.0/24  |


As expected, nodes only include the IP defined as primary:

$ oc get nodes -o json | jq '.items[].status.addresses[]'                                                                                                                                                                            
{
  "address": "172.16.0.67",
  "type": "InternalIP"
}
{
  "address": "ostest-f97g7-master-0",
  "type": "Hostname"
}
{
  "address": "172.16.0.50",
  "type": "InternalIP"
}
{
  "address": "ostest-f97g7-master-1",
  "type": "Hostname"
}
{
  "address": "172.16.0.87",
  "type": "InternalIP"
}
{
  "address": "ostest-f97g7-master-2",
  "type": "Hostname"
}
{
  "address": "172.16.0.59",
  "type": "InternalIP"
}
{
  "address": "ostest-f97g7-worker-0-5m8x2",
  "type": "Hostname"
}
{
  "address": "172.16.0.210",
  "type": "InternalIP"
}
{
  "address": "ostest-f97g7-worker-0-7tzcq",
  "type": "Hostname"
}
{
  "address": "172.16.0.203",
  "type": "InternalIP"
}
{
  "address": "ostest-f97g7-worker-0-rh8x8",
  "type": "Hostname"
}

and the openstack instances have the two interfaces attached:

$ openstack server list -c Name -c Networks
+-----------------------------+-----------------------------------------------------------------------------------+
| Name                        | Networks                                                                          |
+-----------------------------+-----------------------------------------------------------------------------------+
| ostest-f97g7-worker-0-7tzcq | StorageNFS=172.17.5.220; restricted_network=172.16.0.210                          |
| ostest-f97g7-worker-0-5m8x2 | StorageNFS=172.17.5.175; restricted_network=172.16.0.59                           |
| ostest-f97g7-worker-0-rh8x8 | StorageNFS=172.17.5.162; restricted_network=172.16.0.203                          |
| ostest-f97g7-master-2       | restricted_network=172.16.0.87                                                    |
| ostest-f97g7-master-1       | restricted_network=172.16.0.50                                                    |
| ostest-f97g7-master-0       | restricted_network=172.16.0.67                                                    |
| installer_host              | installer_host-network=172.16.40.208, 10.46.44.182; restricted_network=172.16.0.3 |
+-----------------------------+-----------------------------------------------------------------------------------+

the cluster is fully operational, as observed running the tests.

The same is working with all the networkTypes: OpenShiftSDN, Kuryr and OVNKubernetes.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069