Description of problem: On some HW, the installation fails due to the startup order of operators that caused by a delay in OSP components. Debug info below Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Deploy OSP13 2. Install OCP 4.2 with Kuryr 3. Actual results: installation failed Expected results: Installation passed Additional info: (undercloud) [stack@undercloud-0 ~]$ oc get nodes NAME STATUS ROLES AGE VERSION ostest-wg9bj-master-0 Ready master 12h v1.14.6+31a56cf75 ostest-wg9bj-master-1 Ready master 12h v1.14.6+31a56cf75 ostest-wg9bj-master-2 Ready master 12h v1.14.6+31a56cf75 ostest-wg9bj-worker-cmnlk Ready worker 11h v1.14.6+31a56cf75 ostest-wg9bj-worker-r4sj5 Ready worker 11h v1.14.6+31a56cf75 ostest-wg9bj-worker-wdmxw Ready worker 11h v1.14.6+31a56cf75 (undercloud) [stack@undercloud-0 ~]$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE cloud-credential 4.2.0-0.nightly-2019-11-26-164052 True False False 11h dns 4.2.0-0.nightly-2019-11-26-164052 True False False 11h insights 4.2.0-0.nightly-2019-11-26-164052 True False True 11h kube-apiserver 4.2.0-0.nightly-2019-11-26-164052 True False False 11h kube-controller-manager 4.2.0-0.nightly-2019-11-26-164052 True False False 11h kube-scheduler 4.2.0-0.nightly-2019-11-26-164052 True False False 11h machine-api 4.2.0-0.nightly-2019-11-26-164052 True False False 11h machine-config 4.2.0-0.nightly-2019-11-26-164052 True False False 11h network 4.2.0-0.nightly-2019-11-26-164052 True False False 11h openshift-controller-manager 4.2.0-0.nightly-2019-11-26-164052 True False False 11h operator-lifecycle-manager 4.2.0-0.nightly-2019-11-26-164052 True False False 11h operator-lifecycle-manager-catalog 4.2.0-0.nightly-2019-11-26-164052 True False False 11h operator-lifecycle-manager-packageserver 4.2.0-0.nightly-2019-11-26-164052 True False False 11h service-ca 4.2.0-0.nightly-2019-11-26-164052 True False False 11h (undercloud) [stack@undercloud-0 ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version False True 12h Unable to apply 4.2.0-0.nightly-2019-11-26-164052: an unknown error has occurred (undercloud) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | 981b5fdd-54f8-42ce-83a4-439dcbb062aa | controller-1 | ACTIVE | ctlplane=192.168.24.20 | overcloud-full | controller | | 3e092d44-4de0-49bf-b348-5f0cb5642378 | controller-2 | ACTIVE | ctlplane=192.168.24.17 | overcloud-full | controller | | 694d4d76-45e1-4a4c-a02e-acd69d770b1d | compute-0 | ACTIVE | ctlplane=192.168.24.15 | overcloud-full | compute | | 6e62763e-1fbf-4254-9961-4416bec7d04a | controller-0 | ACTIVE | ctlplane=192.168.24.12 | overcloud-full | controller | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ (undercloud) [stack@undercloud-0 ~]$ . shiftstackrc (shiftstack) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+---------------------------+--------+------------------------------------+-------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+---------------------------+--------+------------------------------------+-------+-----------+ | c343db2e-5941-4e0e-9b0a-c510031d7dce | ostest-wg9bj-worker-r4sj5 | ACTIVE | ostest-wg9bj-openshift=10.196.0.36 | rhcos | m4.xlarge | | ae93ffae-db0c-4282-9a76-e5e1dca97e86 | ostest-wg9bj-worker-wdmxw | ACTIVE | ostest-wg9bj-openshift=10.196.0.38 | rhcos | m4.xlarge | | af9784e1-e6f5-4f44-8058-b83df6c54d40 | ostest-wg9bj-worker-cmnlk | ACTIVE | ostest-wg9bj-openshift=10.196.0.34 | rhcos | m4.xlarge | | f2a8fc24-78cc-44bf-95f4-a94775d15c63 | ostest-wg9bj-master-0 | ACTIVE | ostest-wg9bj-openshift=10.196.0.13 | rhcos | m4.xlarge | | 719a23bc-85f4-42c1-aea2-a9afc62d95a4 | ostest-wg9bj-master-2 | ACTIVE | ostest-wg9bj-openshift=10.196.0.29 | rhcos | m4.xlarge | | dd693a11-4fee-4149-a3c2-4c8717085f2d | ostest-wg9bj-master-1 | ACTIVE | ostest-wg9bj-openshift=10.196.0.15 | rhcos | m4.xlarge | +--------------------------------------+---------------------------+--------+------------------------------------+-------+-----------+ (shiftstack) [stack@undercloud-0 ~]$ oc describe pod openshift-apiserver-operator-7648756bf6-snj67 -n openshift-apiserver-operator Name: openshift-apiserver-operator-7648756bf6-snj67 Namespace: openshift-apiserver-operator Priority: 2000000000 PriorityClassName: system-cluster-critical Node: ostest-wg9bj-master-1/10.196.0.15 Start Time: Tue, 26 Nov 2019 17:20:01 -0500 Labels: app=openshift-apiserver-operator pod-template-hash=7648756bf6 Annotations: k8s.v1.cni.cncf.io/networks-status: [{ "name": "kuryr", "interface": "eth0", "ips": [ "10.128.40.9" ], "mac": "fa:16:3e:b9:fd:e6", "default": true, "dns": {} }] openstack.org/kuryr-pod-label: {"app": "openshift-apiserver-operator", "pod-template-hash": "7648756bf6"} openstack.org/kuryr-vif: {"versioned_object.changes": ["default_vif"], "versioned_object.data": {"additional_vifs": {}, "default_vif": {"versioned_object.changes":... Status: Running IP: 10.128.40.9 Controlled By: ReplicaSet/openshift-apiserver-operator-7648756bf6 Containers: openshift-apiserver-operator: Container ID: cri-o://84e9889eda25003036998b2c0bbd5d2d06ca74f5f62a690a40b4601887f10f03 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7 Port: 8443/TCP Host Port: 0/TCP Command: cluster-openshift-apiserver-operator operator Args: --config=/var/run/configmaps/config/config.yaml State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Message: I1127 10:19:30.592561 1 cmd.go:160] Using service-serving-cert provided certificates I1127 10:19:30.593211 1 observer_polling.go:106] Starting file observer W1127 10:19:33.664944 1 builder.go:181] unable to get owner reference (falling back to namespace): Get https://172.30.0.1:443/api/v1/namespaces/openshift-apiserver-operator/pods: dial tcp 172.30.0.1:443: connect: no route to host F1127 10:20:04.555120 1 cmd.go:92] Get https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 172.30.0.1:443: connect: no route to host Exit Code: 255 Started: Wed, 27 Nov 2019 05:19:30 -0500 Finished: Wed, 27 Nov 2019 05:20:04 -0500 Ready: False Restart Count: 130 Requests: cpu: 10m memory: 50Mi Environment: IMAGE: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:699ef43532027b2429eeb48b809e27c2c1b0e750b2e5fa0d84159540c141c7ca OPERATOR_IMAGE: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7 OPERATOR_IMAGE_VERSION: 4.2.0-0.nightly-2019-11-26-164052 OPERAND_IMAGE_VERSION: 4.2.0-0.nightly-2019-11-26-164052 Mounts: /var/run/configmaps/config from config (rw) /var/run/secrets/kubernetes.io/serviceaccount from openshift-apiserver-operator-token-scnqv (ro) /var/run/secrets/serving-cert from serving-cert (rw) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: serving-cert: Type: Secret (a volume populated by a Secret) SecretName: openshift-apiserver-operator-serving-cert Optional: true config: Type: ConfigMap (a volume populated by a ConfigMap) Name: openshift-apiserver-operator-config Optional: false openshift-apiserver-operator-token-scnqv: Type: Secret (a volume populated by a Secret) SecretName: openshift-apiserver-operator-token-scnqv Optional: false QoS Class: Burstable Node-Selectors: node-role.kubernetes.io/master= Tolerations: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/not-ready:NoExecute for 120s node.kubernetes.io/unreachable:NoExecute for 120s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 24m (x126 over 11h) kubelet, ostest-wg9bj-master-1 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7" already present on machine Warning BackOff 4m19s (x2946 over 11h) kubelet, ostest-wg9bj-master-1 Back-off restarting failed container (shiftstack) [stack@undercloud-0 ~]$ #oc describe pod openshift-apiserver-operator-7648756bf6-snj67 -n openshift-apiserver-operator (shiftstack) [stack@undercloud-0 ~]$ oc logs openshift-apiserver-operator-7648756bf6-snj67 -n openshift-apiserver-operator I1127 10:19:30.592561 1 cmd.go:160] Using service-serving-cert provided certificates I1127 10:19:30.593211 1 observer_polling.go:106] Starting file observer W1127 10:19:33.664944 1 builder.go:181] unable to get owner reference (falling back to namespace): Get https://172.30.0.1:443/api/v1/namespaces/openshift-apiserver-operator/pods: dial tcp 172.30.0.1:443: connect: no route to host F1127 10:20:04.555120 1 cmd.go:92] Get https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 172.30.0.1:443: connect: no route to host (shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer list +--------------------------------------+-----------------------------------------------------------------------+----------------------------------+----------------+---------------------+----------+ | id | name | project_id | vip_address | provisioning_status | provider | +--------------------------------------+-----------------------------------------------------------------------+----------------------------------+----------------+---------------------+----------+ | 9f6f2c5f-4fdd-4986-8d39-1d59768206ef | ostest-wg9bj-kuryr-api-loadbalancer | aa0378b6eb544d59a84c78a790f0a768 | 172.30.0.1 | ACTIVE | octavia | | 07489b31-730e-4f2d-b7e8-83c8b6888ad2 | openshift-etcd/etcd | aa0378b6eb544d59a84c78a790f0a768 | 172.30.20.213 | ACTIVE | octavia | | 28c0891a-addb-4d34-9221-83d2cd36b985 | openshift-cluster-version/cluster-version-operator | aa0378b6eb544d59a84c78a790f0a768 | 172.30.118.219 | ACTIVE | octavia | | 9ce335fb-2565-447a-bdbb-a7be2d3f6769 | openshift-kube-scheduler-operator/metrics | aa0378b6eb544d59a84c78a790f0a768 | 172.30.130.117 | ACTIVE | octavia | | 6ef04bb2-4378-43aa-b793-1f3dc2a177a6 | openshift-kube-controller-manager-operator/metrics | aa0378b6eb544d59a84c78a790f0a768 | 172.30.62.162 | ACTIVE | octavia | | 1b1a1ffe-7048-457c-8e92-c8b53be2a30d | openshift-kuryr/kuryr-dns-admission-controller | aa0378b6eb544d59a84c78a790f0a768 | 172.30.62.33 | ACTIVE | octavia | | 314956c5-e8f7-4505-8118-eb9c664a45d0 | openshift-machine-api/machine-api-operator | aa0378b6eb544d59a84c78a790f0a768 | 172.30.82.129 | ACTIVE | octavia | | f35fa438-d40d-4454-ba00-046e74ad4b72 | openshift-apiserver-operator/metrics | aa0378b6eb544d59a84c78a790f0a768 | 172.30.54.49 | ACTIVE | octavia | | dfc5427c-89ca-4af1-b8c6-43d9ea6b8869 | openshift-cloud-credential-operator/controller-manager-service | aa0378b6eb544d59a84c78a790f0a768 | 172.30.121.201 | ACTIVE | octavia | | 46c4535d-ef50-473f-add5-1d7613639185 | openshift-kube-apiserver-operator/metrics | aa0378b6eb544d59a84c78a790f0a768 | 172.30.211.35 | ACTIVE | octavia | | 854da7fc-d095-459e-a1b8-3e7de3a36bd0 | openshift-controller-manager-operator/metrics | aa0378b6eb544d59a84c78a790f0a768 | 172.30.56.92 | ACTIVE | octavia | | dbbae311-14c3-4298-a228-4df5e500275e | openshift-dns/dns-default | aa0378b6eb544d59a84c78a790f0a768 | 172.30.0.10 | ACTIVE | octavia | | 74bd7e77-97ff-4e06-89c7-f6ed68e1a988 | openshift-multus/multus-admission-controller | aa0378b6eb544d59a84c78a790f0a768 | 172.30.82.85 | ACTIVE | octavia | | 307b0eb5-ee04-467a-a548-fde874cd95a6 | openshift-kube-apiserver/apiserver | aa0378b6eb544d59a84c78a790f0a768 | 172.30.194.43 | ACTIVE | octavia | | 280204d9-5cfb-4acf-8983-1737dae5d773 | openshift-operator-lifecycle-manager/olm-operator-metrics | aa0378b6eb544d59a84c78a790f0a768 | 172.30.248.45 | ACTIVE | octavia | | 9ad9135f-0663-44fb-a984-640c0e8f639e | openshift-kube-scheduler/scheduler | aa0378b6eb544d59a84c78a790f0a768 | 172.30.226.138 | ACTIVE | octavia | | 646a2ead-525f-48b2-9983-9c306b0745b1 | openshift-operator-lifecycle-manager/catalog-operator-metrics | aa0378b6eb544d59a84c78a790f0a768 | 172.30.17.72 | ACTIVE | octavia | | 1bcbd619-251f-4043-8ea2-c8dd23760dae | openshift-controller-manager/controller-manager | aa0378b6eb544d59a84c78a790f0a768 | 172.30.62.44 | ACTIVE | octavia | | e273fc88-8e1e-432c-9072-4acea769549c | openshift-kube-controller-manager/kube-controller-manager | aa0378b6eb544d59a84c78a790f0a768 | 172.30.32.62 | ACTIVE | octavia | | d035e23a-40fe-462c-af3a-98b553454e0f | openshift-operator-lifecycle-manager/v1-packages-operators-coreos-com | aa0378b6eb544d59a84c78a790f0a768 | 172.30.16.122 | ACTIVE | octavia | +--------------------------------------+-----------------------------------------------------------------------+----------------------------------+----------------+---------------------+----------+ (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list +--------------------------------------+--------------------------------------+-----------+------------+---------------+----------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+------------+---------------+----------------+ | 09453a6a-b75e-4100-a909-b0a5bf4ac6f2 | 9f6f2c5f-4fdd-4986-8d39-1d59768206ef | ALLOCATED | STANDALONE | 172.24.0.27 | 172.30.0.1 | | 158a48ab-867b-458f-94e3-6aba65609996 | d035e23a-40fe-462c-af3a-98b553454e0f | ALLOCATED | STANDALONE | 172.24.0.30 | 172.30.16.122 | | 1cbfdbf2-0004-4bcc-a72a-dcde7bc61b26 | f35fa438-d40d-4454-ba00-046e74ad4b72 | ALLOCATED | STANDALONE | 172.24.0.14 | 172.30.54.49 | | 25e45a15-fe95-4987-a390-3181d7a008ad | 9ce335fb-2565-447a-bdbb-a7be2d3f6769 | ALLOCATED | STANDALONE | 172.24.0.6 | 172.30.130.117 | | 326c93b5-dec0-4821-bbb1-0385a81a7f07 | 6ef04bb2-4378-43aa-b793-1f3dc2a177a6 | ALLOCATED | STANDALONE | 172.24.0.17 | 172.30.62.162 | | 3a9fac66-8751-4219-9133-dd86e02ae34f | dfc5427c-89ca-4af1-b8c6-43d9ea6b8869 | ALLOCATED | STANDALONE | 172.24.0.33 | 172.30.121.201 | | 50635b26-d9b3-4377-b384-d170e11b92be | e273fc88-8e1e-432c-9072-4acea769549c | ALLOCATED | STANDALONE | 172.24.0.21 | 172.30.32.62 | | 5c2f3b69-f614-4ee0-ba00-d394488bea4b | 9ad9135f-0663-44fb-a984-640c0e8f639e | ALLOCATED | STANDALONE | 172.24.0.20 | 172.30.226.138 | | 5ff184ec-5164-4d81-9080-436c955f1797 | 314956c5-e8f7-4505-8118-eb9c664a45d0 | ALLOCATED | STANDALONE | 172.24.0.9 | 172.30.82.129 | | 61e0968a-d4ea-4bd2-bdc8-b19c99f0f844 | 74bd7e77-97ff-4e06-89c7-f6ed68e1a988 | ALLOCATED | STANDALONE | 172.24.0.19 | 172.30.82.85 | | 7264d58e-c2a1-4efe-b8c2-b6145bdea831 | dbbae311-14c3-4298-a228-4df5e500275e | ALLOCATED | STANDALONE | 172.24.0.16 | 172.30.0.10 | | 7b11dc69-71a3-43c7-b680-66150134ce65 | 07489b31-730e-4f2d-b7e8-83c8b6888ad2 | ALLOCATED | STANDALONE | 172.24.0.11 | 172.30.20.213 | | aa3c623d-be46-40fa-9731-c528ad1a744e | 307b0eb5-ee04-467a-a548-fde874cd95a6 | ALLOCATED | STANDALONE | 172.24.0.22 | 172.30.194.43 | | c3bc608d-a70a-4b14-940c-4c2ca167b93f | 646a2ead-525f-48b2-9983-9c306b0745b1 | ALLOCATED | STANDALONE | 172.24.0.12 | 172.30.17.72 | | d1ea9e02-2914-4af4-9e64-f97624d30113 | 46c4535d-ef50-473f-add5-1d7613639185 | ALLOCATED | STANDALONE | 172.24.0.38 | 172.30.211.35 | | d2d2727d-e16d-4a28-b85a-3d1406008fbc | 1b1a1ffe-7048-457c-8e92-c8b53be2a30d | ALLOCATED | STANDALONE | 172.24.0.13 | 172.30.62.33 | | f485fdbd-b0a0-4f64-b578-a1e2b4b18c33 | 28c0891a-addb-4d34-9221-83d2cd36b985 | ALLOCATED | STANDALONE | 172.24.0.15 | 172.30.118.219 | | f52cf9f9-02ee-4a8c-9451-4c4e24f7d629 | 1bcbd619-251f-4043-8ea2-c8dd23760dae | ALLOCATED | STANDALONE | 172.24.0.42 | 172.30.62.44 | | f90b5b9f-7dce-4a9d-859c-d414e0499088 | 854da7fc-d095-459e-a1b8-3e7de3a36bd0 | ALLOCATED | STANDALONE | 172.24.0.24 | 172.30.56.92 | | fa6ba4d4-0df5-4934-a10c-175e75b0743c | 280204d9-5cfb-4acf-8983-1737dae5d773 | ALLOCATED | STANDALONE | 172.24.0.10 | 172.30.248.45 | +--------------------------------------+--------------------------------------+-----------+------------+---------------+----------------+ (overcloud) [stack@undercloud-0 ~]$ oc get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE openshift-apiserver-operator openshift-apiserver-operator-7648756bf6-snj67 1/1 Running 131 12h openshift-cloud-credential-operator cloud-credential-operator-cd56bbfd6-kgp9b 1/1 Running 1 12h openshift-cluster-machine-approver machine-approver-f99ddd99f-bst6c 1/1 Running 0 12h openshift-cluster-version cluster-version-operator-865b5b8d46-gf2j4 1/1 Running 1 12h openshift-controller-manager-operator openshift-controller-manager-operator-58f688fbd9-kvxg2 1/1 Running 2 12h openshift-controller-manager controller-manager-fhzxf 1/1 Running 35 12h openshift-controller-manager controller-manager-r7xtx 1/1 Running 40 12h openshift-controller-manager controller-manager-wrf2b 1/1 Running 49 12h openshift-dns-operator dns-operator-d864b4d65-5g68h 1/1 Running 0 12h openshift-dns dns-default-j7n9t 2/2 Running 0 11h openshift-dns dns-default-krv6q 2/2 Running 0 12h openshift-dns dns-default-m6k72 2/2 Running 0 11h openshift-dns dns-default-sgxs7 2/2 Running 0 12h openshift-dns dns-default-vl9t7 2/2 Running 0 12h openshift-dns dns-default-w65sk 2/2 Running 0 11h openshift-etcd etcd-member-ostest-wg9bj-master-0 2/2 Running 0 12h openshift-etcd etcd-member-ostest-wg9bj-master-1 2/2 Running 0 12h openshift-etcd etcd-member-ostest-wg9bj-master-2 2/2 Running 0 12h openshift-insights insights-operator-5568799bf7-wzpv6 1/1 Running 0 12h openshift-kube-apiserver-operator kube-apiserver-operator-dc675f755-h5drq 1/1 Running 2 12h openshift-kube-apiserver installer-2-ostest-wg9bj-master-0 0/1 Completed 0 11h openshift-kube-apiserver installer-2-ostest-wg9bj-master-1 0/1 Completed 0 12h openshift-kube-apiserver installer-2-ostest-wg9bj-master-2 0/1 Completed 0 12h openshift-kube-apiserver installer-3-ostest-wg9bj-master-0 0/1 Completed 0 11h openshift-kube-apiserver installer-3-ostest-wg9bj-master-1 0/1 Completed 0 11h openshift-kube-apiserver installer-3-ostest-wg9bj-master-2 0/1 Completed 0 11h openshift-kube-apiserver kube-apiserver-ostest-wg9bj-master-0 3/3 Running 0 11h openshift-kube-apiserver kube-apiserver-ostest-wg9bj-master-1 3/3 Running 0 11h openshift-kube-apiserver kube-apiserver-ostest-wg9bj-master-2 3/3 Running 0 11h openshift-kube-apiserver revision-pruner-2-ostest-wg9bj-master-0 0/1 Completed 0 11h openshift-kube-apiserver revision-pruner-2-ostest-wg9bj-master-1 0/1 Completed 0 11h openshift-kube-apiserver revision-pruner-2-ostest-wg9bj-master-2 0/1 Completed 0 11h openshift-kube-apiserver revision-pruner-3-ostest-wg9bj-master-0 0/1 Completed 0 11h openshift-kube-apiserver revision-pruner-3-ostest-wg9bj-master-1 0/1 Completed 0 11h openshift-kube-apiserver revision-pruner-3-ostest-wg9bj-master-2 0/1 Completed 0 11h openshift-kube-controller-manager-operator kube-controller-manager-operator-586f864f89-crr88 1/1 Running 2 12h openshift-kube-controller-manager installer-2-ostest-wg9bj-master-0 0/1 Completed 0 12h openshift-kube-controller-manager installer-6-ostest-wg9bj-master-0 0/1 Completed 0 11h openshift-kube-controller-manager installer-6-ostest-wg9bj-master-1 0/1 Completed 0 12h openshift-kube-controller-manager installer-6-ostest-wg9bj-master-2 0/1 Completed 0 11h openshift-kube-controller-manager installer-7-ostest-wg9bj-master-0 0/1 Completed 0 11h openshift-kube-controller-manager installer-7-ostest-wg9bj-master-1 0/1 Completed 0 11h openshift-kube-controller-manager installer-7-ostest-wg9bj-master-2 0/1 Completed 0 11h openshift-kube-controller-manager kube-controller-manager-ostest-wg9bj-master-0 2/2 Running 1 11h openshift-kube-controller-manager kube-controller-manager-ostest-wg9bj-master-1 2/2 Running 0 11h openshift-kube-controller-manager kube-controller-manager-ostest-wg9bj-master-2 2/2 Running 1 11h openshift-kube-controller-manager revision-pruner-2-ostest-wg9bj-master-0 0/1 Completed 0 12h openshift-kube-controller-manager revision-pruner-6-ostest-wg9bj-master-0 0/1 Completed 0 11h openshift-kube-controller-manager revision-pruner-6-ostest-wg9bj-master-1 0/1 Completed 0 11h openshift-kube-controller-manager revision-pruner-6-ostest-wg9bj-master-2 0/1 Completed 0 11h openshift-kube-controller-manager revision-pruner-7-ostest-wg9bj-master-0 0/1 Completed 0 11h openshift-kube-controller-manager revision-pruner-7-ostest-wg9bj-master-1 0/1 Completed 0 11h openshift-kube-controller-manager revision-pruner-7-ostest-wg9bj-master-2 0/1 Completed 0 11h openshift-kube-scheduler-operator openshift-kube-scheduler-operator-644d57c785-wqtxg 1/1 Running 2 12h openshift-kube-scheduler installer-2-ostest-wg9bj-master-2 0/1 Completed 0 12h openshift-kube-scheduler installer-4-ostest-wg9bj-master-0 0/1 Completed 0 12h openshift-kube-scheduler installer-4-ostest-wg9bj-master-1 0/1 Completed 0 11h openshift-kube-scheduler installer-4-ostest-wg9bj-master-2 0/1 Completed 0 11h openshift-kube-scheduler openshift-kube-scheduler-ostest-wg9bj-master-0 1/1 Running 0 12h openshift-kube-scheduler openshift-kube-scheduler-ostest-wg9bj-master-1 1/1 Running 1 11h openshift-kube-scheduler openshift-kube-scheduler-ostest-wg9bj-master-2 1/1 Running 1 11h openshift-kube-scheduler revision-pruner-2-ostest-wg9bj-master-2 0/1 Completed 0 12h openshift-kube-scheduler revision-pruner-4-ostest-wg9bj-master-0 0/1 Completed 0 11h openshift-kube-scheduler revision-pruner-4-ostest-wg9bj-master-1 0/1 Completed 0 11h openshift-kube-scheduler revision-pruner-4-ostest-wg9bj-master-2 0/1 Completed 0 11h openshift-kuryr kuryr-cni-48f6d 1/1 Running 0 12h openshift-kuryr kuryr-cni-55hw8 1/1 Running 0 12h openshift-kuryr kuryr-cni-6swr7 1/1 Running 0 12h openshift-kuryr kuryr-cni-dqdc7 1/1 Running 0 11h openshift-kuryr kuryr-cni-gszvd 1/1 Running 0 11h openshift-kuryr kuryr-cni-z72wc 1/1 Running 0 11h openshift-kuryr kuryr-controller-8557787669-ldqj5 1/1 Running 3 12h openshift-kuryr kuryr-dns-admission-controller-8wx5l 1/1 Running 0 12h openshift-kuryr kuryr-dns-admission-controller-lqdq6 1/1 Running 0 12h openshift-kuryr kuryr-dns-admission-controller-z2zbp 1/1 Running 0 12h openshift-machine-api machine-api-controllers-7b9c567969-rw52l 3/3 Running 0 12h openshift-machine-api machine-api-operator-668d945596-j2xvc 1/1 Running 1 12h openshift-machine-config-operator etcd-quorum-guard-74fbfccd87-2mrgb 1/1 Running 0 12h openshift-machine-config-operator etcd-quorum-guard-74fbfccd87-7qgkk 1/1 Running 0 12h openshift-machine-config-operator etcd-quorum-guard-74fbfccd87-hmtqn 1/1 Running 0 12h openshift-machine-config-operator machine-config-controller-c97d65bd7-9zt4t 1/1 Running 1 12h openshift-machine-config-operator machine-config-daemon-c648v 1/1 Running 0 11h openshift-machine-config-operator machine-config-daemon-hrcn6 1/1 Running 0 12h openshift-machine-config-operator machine-config-daemon-kkgft 1/1 Running 0 11h openshift-machine-config-operator machine-config-daemon-ng22z 1/1 Running 0 11h openshift-machine-config-operator machine-config-daemon-nwjs6 1/1 Running 0 12h openshift-machine-config-operator machine-config-daemon-sdpsp 1/1 Running 0 12h openshift-machine-config-operator machine-config-operator-8467996985-w7skh 1/1 Running 1 12h openshift-machine-config-operator machine-config-server-89d5w 1/1 Running 0 12h openshift-machine-config-operator machine-config-server-cbhhv 1/1 Running 0 12h openshift-machine-config-operator machine-config-server-xjxfx 1/1 Running 0 12h openshift-multus multus-2qfng 1/1 Running 0 11h openshift-multus multus-8zgpl 1/1 Running 0 11h openshift-multus multus-admission-controller-2zbgt 1/1 Running 0 12h openshift-multus multus-admission-controller-94jtk 1/1 Running 0 12h openshift-multus multus-admission-controller-z5pjg 1/1 Running 0 12h openshift-multus multus-h7h27 1/1 Running 0 12h openshift-multus multus-hf87d 1/1 Running 0 12h openshift-multus multus-nc9vz 1/1 Running 0 12h openshift-multus multus-z5sgz 1/1 Running 0 11h openshift-network-operator network-operator-7fdc49495d-44nfj 1/1 Running 0 12h openshift-openstack-infra coredns-ostest-wg9bj-master-0 1/1 Running 0 12h openshift-openstack-infra coredns-ostest-wg9bj-master-1 1/1 Running 0 12h openshift-openstack-infra coredns-ostest-wg9bj-master-2 1/1 Running 0 12h openshift-openstack-infra coredns-ostest-wg9bj-worker-cmnlk 1/1 Running 0 11h openshift-openstack-infra coredns-ostest-wg9bj-worker-r4sj5 1/1 Running 0 11h openshift-openstack-infra coredns-ostest-wg9bj-worker-wdmxw 1/1 Running 0 11h openshift-openstack-infra haproxy-ostest-wg9bj-master-0 2/2 Running 1 12h openshift-openstack-infra haproxy-ostest-wg9bj-master-1 2/2 Running 1 12h openshift-openstack-infra haproxy-ostest-wg9bj-master-2 2/2 Running 1 12h openshift-openstack-infra keepalived-ostest-wg9bj-master-0 1/1 Running 0 12h openshift-openstack-infra keepalived-ostest-wg9bj-master-1 1/1 Running 0 12h openshift-openstack-infra keepalived-ostest-wg9bj-master-2 1/1 Running 0 12h openshift-openstack-infra keepalived-ostest-wg9bj-worker-cmnlk 1/1 Running 0 11h openshift-openstack-infra keepalived-ostest-wg9bj-worker-r4sj5 1/1 Running 0 11h openshift-openstack-infra keepalived-ostest-wg9bj-worker-wdmxw 1/1 Running 0 11h openshift-openstack-infra mdns-publisher-ostest-wg9bj-master-0 1/1 Running 0 12h openshift-openstack-infra mdns-publisher-ostest-wg9bj-master-1 1/1 Running 0 12h openshift-openstack-infra mdns-publisher-ostest-wg9bj-master-2 1/1 Running 0 12h openshift-openstack-infra mdns-publisher-ostest-wg9bj-worker-cmnlk 1/1 Running 0 11h openshift-openstack-infra mdns-publisher-ostest-wg9bj-worker-r4sj5 1/1 Running 0 11h openshift-openstack-infra mdns-publisher-ostest-wg9bj-worker-wdmxw 1/1 Running 0 11h openshift-operator-lifecycle-manager catalog-operator-67d65c7774-hk7pk 1/1 Running 0 12h openshift-operator-lifecycle-manager olm-operator-8559ddbb5c-5dxv2 1/1 Running 0 12h openshift-operator-lifecycle-manager packageserver-b5689448f-wdd4v 1/1 Running 0 11h openshift-operator-lifecycle-manager packageserver-b5689448f-whhzk 1/1 Running 0 11h openshift-service-ca-operator service-ca-operator-8655757d55-jz5h6 1/1 Running 1 12h openshift-service-ca apiservice-cabundle-injector-7c65c88bd9-fpm7x 1/1 Running 1 12h openshift-service-ca configmap-cabundle-injector-6864bf47f7-9pfv2 1/1 Running 1 12h openshift-service-ca service-serving-cert-signer-77c9c666c9-gtbsl 1/1 Running 1 12h (shiftstack) [stack@undercloud-0 ~]$ openstack port list | grep 172.30.0.1 | 0abd15ce-2536-452a-b2b9-054421f506d5 | octavia-lb-dbbae311-14c3-4298-a228-4df5e500275e | fa:16:3e:43:73:11 | ip_address='172.30.0.10', subnet_id='d4c54fd6-9550-4eee-b590-a46abe5a721f' | DOWN | | 57d29893-fbb9-4b2a-93e0-6b07ab5dc8e7 | octavia-lb-9f6f2c5f-4fdd-4986-8d39-1d59768206ef | fa:16:3e:2b:58:a8 | ip_address='172.30.0.1', subnet_id='d4c54fd6-9550-4eee-b590-a46abe5a721f' | DOWN |
Pushing to 4.4 since this is not a 4.3 regression. When we have a fix for it, we can talk about doing a back-port.
This turns out to be a bug in neutron when several subnets/networks are created and connected to the same router around the same time. After investigation, the problem is that even though the created subnet is connected to the router, it is not properly configured: # Subnet $ openstack subnet list | grep 10.128.36 | 6926d902-999e-4faf-94f4-7b2c6ca35a14 | ns/openshift-kube-scheduler-operator-subnet | 54f03a0a-e6e5-43d4-912d-3996d5376da6 | 10.128.36.0/23 | # It even has an ACTIVE gw port (10.128.36.1) $ openstack port list | grep 10.128.36. | 6242053e-fd7c-4a5a-8acf-bae6325db33d | | fa:16:3e:9e:a4:26 | ip_address='10.128.36.7', subnet_id='6926d902-999e-4faf-94f4-7b2c6ca35a14' | ACTIVE | | 68ec4c36-e7aa-44f6-b250-fe7169e70bb4 | | fa:16:3e:79:ab:75 | ip_address='10.128.36.9', subnet_id='6926d902-999e-4faf-94f4-7b2c6ca35a14' | ACTIVE | | 79ee6f2c-8c27-4bda-a7be-918911482fd7 | | fa:16:3e:e8:9c:d7 | ip_address='10.128.36.1', subnet_id='6926d902-999e-4faf-94f4-7b2c6ca35a14' | ACTIVE | | e868089c-49e7-448c-af9b-e565d2bdd170 | | fa:16:3e:2d:15:30 | ip_address='10.128.36.5', subnet_id='6926d902-999e-4faf-94f4-7b2c6ca35a14' | ACTIVE | # And the router shows it is connected to it $ openstack router show ostest-rnf5x-external-router | grep 6926d902-999e-4faf-94f4-7b2c6ca35a14 | interfaces_info | [{"subnet_id": "4d21e651-ae15-4196-ac20-35235f641afe", "ip_address": "10.128.12.1", "port_id": "02192a04-3fec-4ea5-b77e-4285135c683a"}, {"subnet_id": "3fb1a51a-01ab-41cf-a036-adbe7f98aafa", "ip_address": "10.128.56.1", "port_id": "03d98330-5b59-4c8c-b75d-7c5c570fa50a"}, {"subnet_id": "10a8c71c-9b00-4fc3-84ca-341279ec35c7", "ip_address": "10.128.0.1", "port_id": "0619ade6-0528-4689-8dd6-b0871e7e17b6"}, {"subnet_id": "02035a8d-b77c-49c4-8e28-4f9f515c52bc", "ip_address": "10.128.48.1", "port_id": "0a3ec172-d3d3-4d51-84ae-28e0acfa4a43"}, {"subnet_id": "3557cc91-8eaf-4eb1-9484-0e69abe34175", "ip_address": "10.128.20.1", "port_id": "11587773-fbc4-4cca-8869-8f47354fba99"}, {"subnet_id": "7d445abd-98e3-4c3c-8d74-4fb5161172f1", "ip_address": "10.128.84.1", "port_id": "11ca52f8-e4ba-49d2-aa5f-c8d6f7a1e65a"}, {"subnet_id": "68c03029-2ba4-41f2-bd1b-fed8a1d971f9", "ip_address": "10.128.50.1", "port_id": "1547deca-4a4c-465c-b3d6-62c419684c61"}, {"subnet_id": "1394e64c-f865-41b3-872c-bc559ea9d128", "ip_address": "10.128.74.1", "port_id": "1f153eff-f024-4da9-aeb2-8f388747eade"}, {"subnet_id": "da19149d-3f28-45cc-ae1b-4a006dd05797", "ip_address": "10.128.72.1", "port_id": "207c52ad-516c-417f-8f6c-37916856ece0"}, {"subnet_id": "62247aec-42c0-4565-ab77-abf3c4319bfb", "ip_address": "10.128.52.1", "port_id": "20996db4-e16b-479c-b437-6f0eb985b322"}, {"subnet_id": "8ad5c87b-be8c-469a-8610-37315d85a363", "ip_address": "10.128.10.1", "port_id": "20f08e2c-f1a8-436b-8380-32f45616d61b"}, {"subnet_id": "eb56f46d-577b-4da4-9e57-33f2a4e0c1cf", "ip_address": "10.128.42.1", "port_id": "27de3cc3-cd6f-4974-aa71-9c0f975dff03"}, {"subnet_id": "e1de8d8a-8ec7-43d4-a94b-6e68ec4564e5", "ip_address": "10.128.70.1", "port_id": "2b704b41-7a6b-4c64-93cf-409575907f2e"}, {"subnet_id": "d84d916a-cad1-4c60-96a2-5d9b954d6ea5", "ip_address": "10.128.30.1", "port_id": "30746d92-bde9-47dc-a546-dc62406ee7f8"}, {"subnet_id": "bdefbeb1-26ea-4711-b511-fc661ef6b9df", "ip_address": "10.128.8.1", "port_id": "32f0f949-d23a-421a-9d70-82c17fc3f429"}, {"subnet_id": "9d8b14aa-7ec5-4ce7-b6ba-53a96fb0f26e", "ip_address": "10.128.28.1", "port_id": "3732f0ef-9cce-4149-8681-afee83a3b245"}, {"subnet_id": "1c0b2865-7c15-4f1f-acd0-acfaf7c44f62", "ip_address": "10.128.46.1", "port_id": "3bd7154f-836f-4cac-bb38-bf3a3c6ff7fe"}, {"subnet_id": "72861c77-5e9c-4c90-9411-b2957f5074ba", "ip_address": "10.128.86.1", "port_id": "3d2ab07c-4675-4caa-b2f8-2c918998db5b"}, {"subnet_id": "3205c5f1-e951-46e6-9b68-e130a3415744", "ip_address": "10.128.14.1", "port_id": "530b8fba-c84e-4841-bb73-3ea7a2ee107b"}, {"subnet_id": "1bd0328a-9dc7-4f09-adfb-542ec43b97f0", "ip_address": "10.128.58.1", "port_id": "5c5e5162-6484-4e5d-93c5-a9029e019fbc"}, {"subnet_id": "6d324273-0e0a-4d19-b528-5c3a3a246ed4", "ip_address": "10.128.92.1", "port_id": "5c7ea132-823b-416c-baab-62766c37b103"}, {"subnet_id": "b7d6fbfd-3416-4f54-a123-4295bdbbcda2", "ip_address": "10.128.80.1", "port_id": "616fdad5-997b-42e2-8e08-7e3ce06cc893"}, {"subnet_id": "c1874895-ad56-46fb-83a8-06783f6191ee", "ip_address": "10.128.38.1", "port_id": "7421cd39-9e3e-4b7e-945c-f979442ead58"}, {"subnet_id": "32464597-9d13-4d8c-a288-9d6e008cc49d", "ip_address": "10.128.82.1", "port_id": "79c05d72-4691-4104-b185-ba28d2f1b9af"}, {"subnet_id": "6926d902-999e-4faf-94f4-7b2c6ca35a14", "ip_address": "10.128.36.1", "port_id": "79ee6f2c-8c27-4bda-a7be-918911482fd7"}, {"subnet_id": "996b3c6d-322c-4314-901f-f2d17841f486", "ip_address": "10.128.32.1", "port_id": "7fed9645-ea8b-404e-94a6-9d31a316c3d4"}, {"subnet_id": "94de1b87-34bb-459c-af08-e4c6b9e0476c", "ip_address": "10.128.90.1", "port_id": "85dede0c-b4b0-4d8a-a188-7820b91ecc20"}, {"subnet_id": "738217f5-55f0-4b23-8e3e-cbca72db1a01", "ip_address": "10.196.0.1", "port_id": "87a6b915-4d9b-4efb-82a1-e4fd999f4f45"}, {"subnet_id": "e36ce6e9-b6c8-44ec-ae7b-7c03566b9b91", "ip_address": "10.128.66.1", "port_id": "882be296-5402-44bd-af18-92ec0e063cde"}, {"subnet_id": "6924e610-b2e3-47f1-b768-76e51f4b94f0", "ip_address": "10.128.54.1", "port_id": "93b65d38-df88-4fb1-b3fb-c6fe42c053be"}, {"subnet_id": "5ac2915a-58c5-47db-9be5-21d23da036ab", "ip_address": "10.128.4.1", "port_id": "98b01ba1-8ae2-41f2-b541-49515d851fdb"}, {"subnet_id": "ff7b620f-a519-45da-8758-d6756d405dfe", "ip_address": "10.128.64.1", "port_id": "9913d8aa-a15b-4216-ac46-b23162c14503"}, {"subnet_id": "c2d91acf-4548-48c5-9813-2529611d1d2c", "ip_address": "10.128.34.1", "port_id": "9cb40889-76b7-4580-bbbf-8d9d6da6700a"}, {"subnet_id": "09170430-8dfb-4622-a4d2-5b91400f5e51", "ip_address": "10.128.88.1", "port_id": "9edca8cc-58dd-49be-b563-f45ca06b9ba0"}, {"subnet_id": "bb441960-4436-4eb9-b571-eddbb3364dab", "ip_address": "172.31.255.254", "port_id": "9ff837c6-8239-45b3-a41e-8d784a8c237d"}, {"subnet_id": "b197dc72-4f4f-484d-ab1f-12b96cdef52d", "ip_address": "10.128.78.1", "port_id": "a436a35b-e23e-4a65-b320-b0f2c1e29bb0"}, {"subnet_id": "5f79b732-ff7a-4b1a-bdfa-a02808e0307e", "ip_address": "10.128.6.1", "port_id": "ad5d1f96-3f44-4e8e-820f-f3509abeb33b"}, {"subnet_id": "bb42e126-a712-4848-934d-23caef0c6b98", "ip_address": "10.128.76.1", "port_id": "b5bdbd16-eaa4-4a0b-a6c4-cd62453e62e3"}, {"subnet_id": "71a9fd17-e8d1-464f-9b56-e6c508f68406", "ip_address": "10.128.62.1", "port_id": "b6896c37-b914-40fe-9706-37ffdcf3888e"}, {"subnet_id": "0d028514-77a7-42c1-90bb-6896eead2775", "ip_address": "10.128.22.1", "port_id": "c3c6a267-e9f2-4dbd-aa47-fbbf0f0fe655"}, {"subnet_id": "4f65263f-b11f-4d94-81f5-bfc46217e08b", "ip_address": "10.128.68.1", "port_id": "c6c7f19b-870c-4d3e-b9fa-240b12468a11"}, {"subnet_id": "db6529c5-c708-47ef-8116-15c402dfb5fd", "ip_address": "10.128.60.1", "port_id": "d74e3d49-a849-48fe-a24c-2b28e05f563b"}, {"subnet_id": "d988cbda-55f3-45f6-8460-9f3e4b4180e7", "ip_address": "10.128.2.1", "port_id": "d84a8ffc-547c-432b-b24f-b776d92446ac"}, {"subnet_id": "0947c7c9-4ff1-47c9-815e-933294c2b823", "ip_address": "10.128.26.1", "port_id": "d8d89031-445b-4961-9cb9-f547923843cb"}, {"subnet_id": "f1dd9ebe-764f-4df6-b8e1-ffc485058437", "ip_address": "10.128.16.1", "port_id": "dd4802c8-bd4e-4952-b3da-79bc8b4e6527"}, {"subnet_id": "1598b795-e21d-487f-811b-c518cbfbe760", "ip_address": "10.128.24.1", "port_id": "df64ae76-42c5-4a2c-8a42-bd4d75a01108"}, {"subnet_id": "7b1ddde7-0261-41a0-af57-89a0747938a1", "ip_address": "10.128.18.1", "port_id": "fab510a2-1dcc-44cc-9ccd-3883302a7bd2"}, {"subnet_id": "9d129f83-d623-4bd9-bf7f-f20564ae3d08", "ip_address": "10.128.44.1", "port_id": "fec72e95-0fe9-4d51-9fbc-08086f4fab23"}] | The ports in this subnet don't have access to the gateway router. I've check the ARP request is reaching the qrouter namespace in the controller, but there is no ARP response. After checking the qr-XXX configuration related to the subnet, it turns out it is missing the IP: 302: qr-79ee6f2c-8c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:e8:9c:d7 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fee8:9cd7/64 scope link valid_lft forever preferred_lft forever And after manually adding it (or by detaching and attaching again the subnet to/from the router), the connectivity starts working: 302: qr-79ee6f2c-8c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:e8:9c:d7 brd ff:ff:ff:ff:ff:ff inet 10.128.36.1/24 scope global qr-79ee6f2c-8c valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fee8:9cd7/64 scope link valid_lft forever preferred_lft forever
*** Bug 1783258 has been marked as a duplicate of this bug. ***
I was debugging it on neutron's side last 2 days. I added some additional debug logs in test env and here is what I found so far. Port which was failed to be configured: +-----------------------+-----------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+-----------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:vnic_type | normal | | created_at | 2019-12-19T08:28:02Z | | description | | | device_id | 60df726a-95d9-4f72-9461-b830c114aa0e | | device_owner | network:ha_router_replicated_interface | | dns_assignment | {"hostname": "host-10-128-10-1", "ip_address": "10.128.10.1", "fqdn": "host-10-128-10-1.shiftstack.com."} | | dns_domain | | | dns_name | | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "b73f874d-faa5-4d0d-8fee-063994ee66e8", "ip_address": "10.128.10.1"} | | id | f448ad14-8d0a-4912-b787-aa5e86b8c357 | | mac_address | fa:16:3e:7f:b3:fa | | name | | | network_id | 256941d3-412a-4822-b5e7-3d9a74b38ffd | | port_security_enabled | False | | project_id | e8b0db94fa684f9c93377c9ee7774677 | | qos_policy_id | | | revision_number | 12 | | security_groups | | | status | ACTIVE | | tags | | | tenant_id | e8b0db94fa684f9c93377c9ee7774677 | | updated_at | 2019-12-19T08:28:53Z | +-----------------------+-----------------------------------------------------------------------------------------------------------+ It's IP address is 10.128.10.1 and it's from subnet: +-------------------+--------------------------------------------------+ | Field | Value | +-------------------+--------------------------------------------------+ | allocation_pools | {"start": "10.128.10.2", "end": "10.128.11.254"} | | cidr | 10.128.10.0/23 | | created_at | 2019-12-19T08:27:55Z | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 10.128.10.1 | | host_routes | | | id | b73f874d-faa5-4d0d-8fee-063994ee66e8 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | ns/openshift-service-ca-operator-subnet | | network_id | 256941d3-412a-4822-b5e7-3d9a74b38ffd | | project_id | e8b0db94fa684f9c93377c9ee7774677 | | revision_number | 1 | | service_types | | | subnetpool_id | 9eb85e53-33d5-4cd2-be60-09df12f7db34 | | tags | openshiftClusterID=ostest-nxqct | | tenant_id | e8b0db94fa684f9c93377c9ee7774677 | | updated_at | 2019-12-19T08:27:59Z | +-------------------+--------------------------------------------------+ Now, lets check neutron L3 agent logs. This port was configured and added to Keepalived's VIPs here: 2019-12-19 08:28:22.632 223011 DEBUG neutron.agent.linux.keepalived [-] SK: adding VIP [fe80::f816:3eff:fe7f:b3fa/64, qr-f448ad14-8d, link] add_vip /usr/lib/python2.7/site-packages/neutron/agent/linux/keepalived.py:20 but L3 agent said that same vip is already added in the vips list: 2019-12-19 08:28:22.633 223011 DEBUG neutron.agent.linux.keepalived [-] VIP [10.128.10.1/23, qr-f448ad14-8d, None] already present in [...] This list contains instances of class KeepalivedVipAddress which is defined in https://github.com/openstack/neutron/blob/stable/queens/neutron/agent/linux/keepalived.py#L85 Same element is element with same IP address. So, it seems that we have already added 10.128.10.1 in vips list. Quick look at this list and we have: [10.128.10.1/23, qr-f3c06bfa-dd, None] So, lets now look for port which starts with f3c06bfa-dd. There is no such port in neutron db anymore: (shiftstack) [stack@undercloud-0 ~]$ neutron port-list | grep f3c06bfa-dd neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. Next step is to look at L3 agent logs again. There we can see that such port was added about 1 minute before our broken port: 2019-12-19 08:27:39.856 223011 DEBUG neutron.agent.l3.router_info [-] appending port {u'allowed_address_pairs': [], u'extra_dhcp_opts': [], u'updated_at': u'2019-12-19T08:27:23Z', u'dns_domain': u'', u'device_owner': u'network:ha_router_replicated_interface', u'revision_number': 7, u'port_security_enabled': False, u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'7a9f9c7c-da57-4666-af5d-de0fd3840a41', u'prefixlen': 23, u'ip_address': u'10.128.10.1'}], u'id': u'f3c06bfa-ddc7-4e4e-a294-7fd03b076531', u'security_groups': [], u'binding:vif_details': {}, u'address_scopes': {u'4': None, u'6': None}, u'binding:vif_type': u'unbound', u'qos_policy_id': None, u'mac_address': u'fa:16:3e:22:ad:ab', u'project_id': u'e8b0db94fa684f9c93377c9ee7774677', u'status': u'DOWN', u'subnets': [{u'dns_nameservers': [], u'ipv6_ra_mode': None, u'gateway_ip': u'10.128.10.1', u'cidr': u'10.128.10.0/23', u'id': u'7a9f9c7c-da57-4666-af5d-de0fd3840a41', u'subnetpool_id': u'9eb85e53-33d5-4cd2-be60-09df12f7db34'}], u'binding:host_id': u'', u'description': u'', u'tags': [], u'dns_assignment': [{u'hostname': u'host-10-128-10-1', u'ip_address': u'10.128.10.1', u'fqdn': u'host-10-128-10-1.shiftstack.com.'}], u'device_id': u'60df726a-95d9-4f72-9461-b830c114aa0e', u'name': u'', u'admin_state_up': True, u'network_id': u'd4f8c742-7553-43bc-aaf8-d167b74a02da', u'dns_name': u'', u'created_at': u'2019-12-19T08:27:20Z', u'mtu': 1450, u'tenant_id': u'e8b0db94fa684f9c93377c9ee7774677', u'extra_subnets': []} to internal_ports cache _process_internal_ports /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:57 and it was in subnet a9f9c7c-da57-4666-af5d-de0fd3840a41 which also don't exists anymore. Next look at L3 agent gives us the answer. This port and this subnet was removed from L3 agent on: 2019-12-19 08:28:23.383 223011 DEBUG neutron.agent.l3.router_info [-] removing port {u'allowed_address_pairs': [], u'extra_dhcp_opts': [], u'updated_at': u'2019-12-19T08:27:23Z', u'dns_domain': u'', u'device_owner': u'network:ha_router_replicated_interface', u'revision_number': 7, u'port_security_enabled': False, u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'7a9f9c7c-da57-4666-af5d-de0fd3840a41', u'prefixlen': 23, u'ip_address': u'10.128.10.1'}], u'id': u'f3c06bfa-ddc7-4e4e-a294-7fd03b076531', u'security_groups': [], u'binding:vif_details': {}, u'address_scopes': {u'4': None, u'6': None}, u'binding:vif_type': u'unbound', u'qos_policy_id': None, u'mac_address': u'fa:16:3e:22:ad:ab', u'project_id': u'e8b0db94fa684f9c93377c9ee7774677', u'status': u'DOWN', u'subnets': [{u'dns_nameservers': [], u'ipv6_ra_mode': None, u'gateway_ip': u'10.128.10.1', u'cidr': u'10.128.10.0/23', u'id': u'7a9f9c7c-da57-4666-af5dde0fd3840a41', u'subnetpool_id': u'9eb85e53-33d5-4cd2-be60-09df12f7db34'}], u'binding:host_id': u'', u'description': u'', u'tags': [], u'dns_assignment': [{u'hostname': u'host-10-128-10-1', u'ip_address': u'10.128.10.1', u'fqdn': u'host10-128-10-1.shiftstack.com.'}], u'device_id': u'60df726a-95d9-4f72-9461-b830c114aa0e', u'name': u'', u'admin_state_up': True, u'network_id': u'd4f8c742-7553-43bc-aaf8-d167b74a02da', u'dns_name': u'', u'created_at': u'2019-12-19T08:27:20Z', u'mtu': 1450, u'tenant_id': u'e8b0db94fa684f9c93377c9ee7774677', u'extra_subnets': []} from internal_ports cache _process_internal_ports /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:591 So to sum up, issue was caused because first port with 10.128.10.1 was added at 8:27:39 on port f3c06bfa-ddc7-4e4e-a294-7fd03b076531 and after that it was trying to add same IP address on different port 'f448ad14-8d0a-4912-b787-aa5e86b8c357' at 8:28:16. After that at 8:28:23 port was removed f3c06bfa-ddc7-4e4e-a294-7fd03b076531 but our IP address wasn't configured anymore on existing port.
The fix is verified on OpenStack-13.0-RHEL-7-20200214.1: 1. Create several subnets from pool 2. Attach subnets to router 3. Detach subnets from router 4. Delete subnets 5. GOTO 1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0770