Bug 1779654 - [IPI] [OSP] [Kuryr] On some HW, the installation fails due to the startup order of operators that caused by a delay in OSP components
Summary: [IPI] [OSP] [Kuryr] On some HW, the installation fails due to the startup ord...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: ---
Assignee: Slawek Kaplonski
QA Contact: Alex Katz
URL:
Whiteboard:
: 1783258 (view as bug list)
Depends On:
Blocks: 1783258 1790416 1790417
TreeView+ depends on / blocked
 
Reported: 2019-12-04 13:03 UTC by Udi Shkalim
Modified: 2020-03-10 11:26 UTC (History)
12 users (show)

Fixed In Version: openstack-neutron-12.1.0-10.el7ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1783258 1790416 1790417 (view as bug list)
Environment:
Last Closed: 2020-03-10 11:26:15 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1857021 0 None None None 2019-12-19 15:01:53 UTC
Red Hat Product Errata RHBA-2020:0770 0 None None None 2020-03-10 11:26:58 UTC

Description Udi Shkalim 2019-12-04 13:03:34 UTC
Description of problem:
On some HW, the installation fails due to the startup order of operators that caused by a delay in OSP components.
Debug info below

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP13
2. Install OCP 4.2 with Kuryr
3.

Actual results:
installation failed

Expected results:
Installation passed

Additional info:
(undercloud) [stack@undercloud-0 ~]$ oc get nodes
NAME                        STATUS   ROLES    AGE   VERSION
ostest-wg9bj-master-0       Ready    master   12h   v1.14.6+31a56cf75
ostest-wg9bj-master-1       Ready    master   12h   v1.14.6+31a56cf75
ostest-wg9bj-master-2       Ready    master   12h   v1.14.6+31a56cf75
ostest-wg9bj-worker-cmnlk   Ready    worker   11h   v1.14.6+31a56cf75
ostest-wg9bj-worker-r4sj5   Ready    worker   11h   v1.14.6+31a56cf75
ostest-wg9bj-worker-wdmxw   Ready    worker   11h   v1.14.6+31a56cf75
(undercloud) [stack@undercloud-0 ~]$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
cloud-credential                           4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
dns                                        4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
insights                                   4.2.0-0.nightly-2019-11-26-164052   True        False         True       11h
kube-apiserver                             4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
kube-controller-manager                    4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
kube-scheduler                             4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
machine-api                                4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
machine-config                             4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
network                                    4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
openshift-controller-manager               4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
operator-lifecycle-manager                 4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
operator-lifecycle-manager-catalog         4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
operator-lifecycle-manager-packageserver   4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
service-ca                                 4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
(undercloud) [stack@undercloud-0 ~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          12h     Unable to apply 4.2.0-0.nightly-2019-11-26-164052: an unknown error has occurred
(undercloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| 981b5fdd-54f8-42ce-83a4-439dcbb062aa | controller-1 | ACTIVE | ctlplane=192.168.24.20 | overcloud-full | controller |
| 3e092d44-4de0-49bf-b348-5f0cb5642378 | controller-2 | ACTIVE | ctlplane=192.168.24.17 | overcloud-full | controller |
| 694d4d76-45e1-4a4c-a02e-acd69d770b1d | compute-0    | ACTIVE | ctlplane=192.168.24.15 | overcloud-full | compute    |
| 6e62763e-1fbf-4254-9961-4416bec7d04a | controller-0 | ACTIVE | ctlplane=192.168.24.12 | overcloud-full | controller |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
(undercloud) [stack@undercloud-0 ~]$ . shiftstackrc
(shiftstack) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+---------------------------+--------+------------------------------------+-------+-----------+
| ID                                   | Name                      | Status | Networks                           | Image | Flavor    |
+--------------------------------------+---------------------------+--------+------------------------------------+-------+-----------+
| c343db2e-5941-4e0e-9b0a-c510031d7dce | ostest-wg9bj-worker-r4sj5 | ACTIVE | ostest-wg9bj-openshift=10.196.0.36 | rhcos | m4.xlarge |
| ae93ffae-db0c-4282-9a76-e5e1dca97e86 | ostest-wg9bj-worker-wdmxw | ACTIVE | ostest-wg9bj-openshift=10.196.0.38 | rhcos | m4.xlarge |
| af9784e1-e6f5-4f44-8058-b83df6c54d40 | ostest-wg9bj-worker-cmnlk | ACTIVE | ostest-wg9bj-openshift=10.196.0.34 | rhcos | m4.xlarge |
| f2a8fc24-78cc-44bf-95f4-a94775d15c63 | ostest-wg9bj-master-0     | ACTIVE | ostest-wg9bj-openshift=10.196.0.13 | rhcos | m4.xlarge |
| 719a23bc-85f4-42c1-aea2-a9afc62d95a4 | ostest-wg9bj-master-2     | ACTIVE | ostest-wg9bj-openshift=10.196.0.29 | rhcos | m4.xlarge |
| dd693a11-4fee-4149-a3c2-4c8717085f2d | ostest-wg9bj-master-1     | ACTIVE | ostest-wg9bj-openshift=10.196.0.15 | rhcos | m4.xlarge |
+--------------------------------------+---------------------------+--------+------------------------------------+-------+-----------+



(shiftstack) [stack@undercloud-0 ~]$ oc describe pod openshift-apiserver-operator-7648756bf6-snj67 -n openshift-apiserver-operator
Name:               openshift-apiserver-operator-7648756bf6-snj67
Namespace:          openshift-apiserver-operator
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               ostest-wg9bj-master-1/10.196.0.15
Start Time:         Tue, 26 Nov 2019 17:20:01 -0500
Labels:             app=openshift-apiserver-operator
                    pod-template-hash=7648756bf6
Annotations:        k8s.v1.cni.cncf.io/networks-status:
                      [{
                          "name": "kuryr",
                          "interface": "eth0",
                          "ips": [
                              "10.128.40.9"
                          ],
                          "mac": "fa:16:3e:b9:fd:e6",
                          "default": true,
                          "dns": {}
                      }]
                    openstack.org/kuryr-pod-label: {"app": "openshift-apiserver-operator", "pod-template-hash": "7648756bf6"}
                    openstack.org/kuryr-vif:
                      {"versioned_object.changes": ["default_vif"], "versioned_object.data": {"additional_vifs": {}, "default_vif": {"versioned_object.changes":...
Status:             Running
IP:                 10.128.40.9
Controlled By:      ReplicaSet/openshift-apiserver-operator-7648756bf6
Containers:
  openshift-apiserver-operator:
    Container ID:  cri-o://84e9889eda25003036998b2c0bbd5d2d06ca74f5f62a690a40b4601887f10f03
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7
    Image ID:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7
    Port:          8443/TCP
    Host Port:     0/TCP
    Command:
      cluster-openshift-apiserver-operator
      operator
    Args:
      --config=/var/run/configmaps/config/config.yaml
    State:       Waiting
      Reason:    CrashLoopBackOff
    Last State:  Terminated
      Reason:    Error
      Message:   I1127 10:19:30.592561       1 cmd.go:160] Using service-serving-cert provided certificates
I1127 10:19:30.593211       1 observer_polling.go:106] Starting file observer
W1127 10:19:33.664944       1 builder.go:181] unable to get owner reference (falling back to namespace): Get https://172.30.0.1:443/api/v1/namespaces/openshift-apiserver-operator/pods: dial tcp 172.30.0.1:443: connect: no route to host
F1127 10:20:04.555120       1 cmd.go:92] Get https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 172.30.0.1:443: connect: no route to host

      Exit Code:    255
      Started:      Wed, 27 Nov 2019 05:19:30 -0500
      Finished:     Wed, 27 Nov 2019 05:20:04 -0500
    Ready:          False
    Restart Count:  130
    Requests:
      cpu:     10m
      memory:  50Mi
    Environment:
      IMAGE:                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:699ef43532027b2429eeb48b809e27c2c1b0e750b2e5fa0d84159540c141c7ca
      OPERATOR_IMAGE:          quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7
      OPERATOR_IMAGE_VERSION:  4.2.0-0.nightly-2019-11-26-164052
      OPERAND_IMAGE_VERSION:   4.2.0-0.nightly-2019-11-26-164052
    Mounts:
      /var/run/configmaps/config from config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from openshift-apiserver-operator-token-scnqv (ro)
      /var/run/secrets/serving-cert from serving-cert (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  serving-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  openshift-apiserver-operator-serving-cert
    Optional:    true
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openshift-apiserver-operator-config
    Optional:  false
  openshift-apiserver-operator-token-scnqv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  openshift-apiserver-operator-token-scnqv
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 120s
                 node.kubernetes.io/unreachable:NoExecute for 120s
Events:
  Type     Reason   Age                     From                            Message
  ----     ------   ----                    ----                            -------
  Normal   Pulled   24m (x126 over 11h)     kubelet, ostest-wg9bj-master-1  Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7" already present on machine
  Warning  BackOff  4m19s (x2946 over 11h)  kubelet, ostest-wg9bj-master-1  Back-off restarting failed container
(shiftstack) [stack@undercloud-0 ~]$ #oc describe pod openshift-apiserver-operator-7648756bf6-snj67 -n openshift-apiserver-operator
(shiftstack) [stack@undercloud-0 ~]$ oc logs openshift-apiserver-operator-7648756bf6-snj67 -n openshift-apiserver-operator
I1127 10:19:30.592561       1 cmd.go:160] Using service-serving-cert provided certificates
I1127 10:19:30.593211       1 observer_polling.go:106] Starting file observer
W1127 10:19:33.664944       1 builder.go:181] unable to get owner reference (falling back to namespace): Get https://172.30.0.1:443/api/v1/namespaces/openshift-apiserver-operator/pods: dial tcp 172.30.0.1:443: connect: no route to host
F1127 10:20:04.555120       1 cmd.go:92] Get https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 172.30.0.1:443: connect: no route to host








(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer list
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+----------------+---------------------+----------+
| id                                   | name                                                                  | project_id                       | vip_address    | provisioning_status | provider |
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+----------------+---------------------+----------+
| 9f6f2c5f-4fdd-4986-8d39-1d59768206ef | ostest-wg9bj-kuryr-api-loadbalancer                                   | aa0378b6eb544d59a84c78a790f0a768 | 172.30.0.1     | ACTIVE              | octavia  |
| 07489b31-730e-4f2d-b7e8-83c8b6888ad2 | openshift-etcd/etcd                                                   | aa0378b6eb544d59a84c78a790f0a768 | 172.30.20.213  | ACTIVE              | octavia  |
| 28c0891a-addb-4d34-9221-83d2cd36b985 | openshift-cluster-version/cluster-version-operator                    | aa0378b6eb544d59a84c78a790f0a768 | 172.30.118.219 | ACTIVE              | octavia  |
| 9ce335fb-2565-447a-bdbb-a7be2d3f6769 | openshift-kube-scheduler-operator/metrics                             | aa0378b6eb544d59a84c78a790f0a768 | 172.30.130.117 | ACTIVE              | octavia  |
| 6ef04bb2-4378-43aa-b793-1f3dc2a177a6 | openshift-kube-controller-manager-operator/metrics                    | aa0378b6eb544d59a84c78a790f0a768 | 172.30.62.162  | ACTIVE              | octavia  |
| 1b1a1ffe-7048-457c-8e92-c8b53be2a30d | openshift-kuryr/kuryr-dns-admission-controller                        | aa0378b6eb544d59a84c78a790f0a768 | 172.30.62.33   | ACTIVE              | octavia  |
| 314956c5-e8f7-4505-8118-eb9c664a45d0 | openshift-machine-api/machine-api-operator                            | aa0378b6eb544d59a84c78a790f0a768 | 172.30.82.129  | ACTIVE              | octavia  |
| f35fa438-d40d-4454-ba00-046e74ad4b72 | openshift-apiserver-operator/metrics                                  | aa0378b6eb544d59a84c78a790f0a768 | 172.30.54.49   | ACTIVE              | octavia  |
| dfc5427c-89ca-4af1-b8c6-43d9ea6b8869 | openshift-cloud-credential-operator/controller-manager-service        | aa0378b6eb544d59a84c78a790f0a768 | 172.30.121.201 | ACTIVE              | octavia  |
| 46c4535d-ef50-473f-add5-1d7613639185 | openshift-kube-apiserver-operator/metrics                             | aa0378b6eb544d59a84c78a790f0a768 | 172.30.211.35  | ACTIVE              | octavia  |
| 854da7fc-d095-459e-a1b8-3e7de3a36bd0 | openshift-controller-manager-operator/metrics                         | aa0378b6eb544d59a84c78a790f0a768 | 172.30.56.92   | ACTIVE              | octavia  |
| dbbae311-14c3-4298-a228-4df5e500275e | openshift-dns/dns-default                                             | aa0378b6eb544d59a84c78a790f0a768 | 172.30.0.10    | ACTIVE              | octavia  |
| 74bd7e77-97ff-4e06-89c7-f6ed68e1a988 | openshift-multus/multus-admission-controller                          | aa0378b6eb544d59a84c78a790f0a768 | 172.30.82.85   | ACTIVE              | octavia  |
| 307b0eb5-ee04-467a-a548-fde874cd95a6 | openshift-kube-apiserver/apiserver                                    | aa0378b6eb544d59a84c78a790f0a768 | 172.30.194.43  | ACTIVE              | octavia  |
| 280204d9-5cfb-4acf-8983-1737dae5d773 | openshift-operator-lifecycle-manager/olm-operator-metrics             | aa0378b6eb544d59a84c78a790f0a768 | 172.30.248.45  | ACTIVE              | octavia  |
| 9ad9135f-0663-44fb-a984-640c0e8f639e | openshift-kube-scheduler/scheduler                                    | aa0378b6eb544d59a84c78a790f0a768 | 172.30.226.138 | ACTIVE              | octavia  |
| 646a2ead-525f-48b2-9983-9c306b0745b1 | openshift-operator-lifecycle-manager/catalog-operator-metrics         | aa0378b6eb544d59a84c78a790f0a768 | 172.30.17.72   | ACTIVE              | octavia  |
| 1bcbd619-251f-4043-8ea2-c8dd23760dae | openshift-controller-manager/controller-manager                       | aa0378b6eb544d59a84c78a790f0a768 | 172.30.62.44   | ACTIVE              | octavia  |
| e273fc88-8e1e-432c-9072-4acea769549c | openshift-kube-controller-manager/kube-controller-manager             | aa0378b6eb544d59a84c78a790f0a768 | 172.30.32.62   | ACTIVE              | octavia  |
| d035e23a-40fe-462c-af3a-98b553454e0f | openshift-operator-lifecycle-manager/v1-packages-operators-coreos-com | aa0378b6eb544d59a84c78a790f0a768 | 172.30.16.122  | ACTIVE              | octavia  |
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+----------------+---------------------+----------+





(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list
+--------------------------------------+--------------------------------------+-----------+------------+---------------+----------------+
| id                                   | loadbalancer_id                      | status    | role       | lb_network_ip | ha_ip          |
+--------------------------------------+--------------------------------------+-----------+------------+---------------+----------------+
| 09453a6a-b75e-4100-a909-b0a5bf4ac6f2 | 9f6f2c5f-4fdd-4986-8d39-1d59768206ef | ALLOCATED | STANDALONE | 172.24.0.27   | 172.30.0.1     |
| 158a48ab-867b-458f-94e3-6aba65609996 | d035e23a-40fe-462c-af3a-98b553454e0f | ALLOCATED | STANDALONE | 172.24.0.30   | 172.30.16.122  |
| 1cbfdbf2-0004-4bcc-a72a-dcde7bc61b26 | f35fa438-d40d-4454-ba00-046e74ad4b72 | ALLOCATED | STANDALONE | 172.24.0.14   | 172.30.54.49   |
| 25e45a15-fe95-4987-a390-3181d7a008ad | 9ce335fb-2565-447a-bdbb-a7be2d3f6769 | ALLOCATED | STANDALONE | 172.24.0.6    | 172.30.130.117 |
| 326c93b5-dec0-4821-bbb1-0385a81a7f07 | 6ef04bb2-4378-43aa-b793-1f3dc2a177a6 | ALLOCATED | STANDALONE | 172.24.0.17   | 172.30.62.162  |
| 3a9fac66-8751-4219-9133-dd86e02ae34f | dfc5427c-89ca-4af1-b8c6-43d9ea6b8869 | ALLOCATED | STANDALONE | 172.24.0.33   | 172.30.121.201 |
| 50635b26-d9b3-4377-b384-d170e11b92be | e273fc88-8e1e-432c-9072-4acea769549c | ALLOCATED | STANDALONE | 172.24.0.21   | 172.30.32.62   |
| 5c2f3b69-f614-4ee0-ba00-d394488bea4b | 9ad9135f-0663-44fb-a984-640c0e8f639e | ALLOCATED | STANDALONE | 172.24.0.20   | 172.30.226.138 |
| 5ff184ec-5164-4d81-9080-436c955f1797 | 314956c5-e8f7-4505-8118-eb9c664a45d0 | ALLOCATED | STANDALONE | 172.24.0.9    | 172.30.82.129  |
| 61e0968a-d4ea-4bd2-bdc8-b19c99f0f844 | 74bd7e77-97ff-4e06-89c7-f6ed68e1a988 | ALLOCATED | STANDALONE | 172.24.0.19   | 172.30.82.85   |
| 7264d58e-c2a1-4efe-b8c2-b6145bdea831 | dbbae311-14c3-4298-a228-4df5e500275e | ALLOCATED | STANDALONE | 172.24.0.16   | 172.30.0.10    |
| 7b11dc69-71a3-43c7-b680-66150134ce65 | 07489b31-730e-4f2d-b7e8-83c8b6888ad2 | ALLOCATED | STANDALONE | 172.24.0.11   | 172.30.20.213  |
| aa3c623d-be46-40fa-9731-c528ad1a744e | 307b0eb5-ee04-467a-a548-fde874cd95a6 | ALLOCATED | STANDALONE | 172.24.0.22   | 172.30.194.43  |
| c3bc608d-a70a-4b14-940c-4c2ca167b93f | 646a2ead-525f-48b2-9983-9c306b0745b1 | ALLOCATED | STANDALONE | 172.24.0.12   | 172.30.17.72   |
| d1ea9e02-2914-4af4-9e64-f97624d30113 | 46c4535d-ef50-473f-add5-1d7613639185 | ALLOCATED | STANDALONE | 172.24.0.38   | 172.30.211.35  |
| d2d2727d-e16d-4a28-b85a-3d1406008fbc | 1b1a1ffe-7048-457c-8e92-c8b53be2a30d | ALLOCATED | STANDALONE | 172.24.0.13   | 172.30.62.33   |
| f485fdbd-b0a0-4f64-b578-a1e2b4b18c33 | 28c0891a-addb-4d34-9221-83d2cd36b985 | ALLOCATED | STANDALONE | 172.24.0.15   | 172.30.118.219 |
| f52cf9f9-02ee-4a8c-9451-4c4e24f7d629 | 1bcbd619-251f-4043-8ea2-c8dd23760dae | ALLOCATED | STANDALONE | 172.24.0.42   | 172.30.62.44   |
| f90b5b9f-7dce-4a9d-859c-d414e0499088 | 854da7fc-d095-459e-a1b8-3e7de3a36bd0 | ALLOCATED | STANDALONE | 172.24.0.24   | 172.30.56.92   |
| fa6ba4d4-0df5-4934-a10c-175e75b0743c | 280204d9-5cfb-4acf-8983-1737dae5d773 | ALLOCATED | STANDALONE | 172.24.0.10   | 172.30.248.45  |
+--------------------------------------+--------------------------------------+-----------+------------+---------------+----------------+





(overcloud) [stack@undercloud-0 ~]$ oc get pods -A
NAMESPACE                                    NAME                                                     READY   STATUS      RESTARTS   AGE
openshift-apiserver-operator                 openshift-apiserver-operator-7648756bf6-snj67            1/1     Running     131        12h
openshift-cloud-credential-operator          cloud-credential-operator-cd56bbfd6-kgp9b                1/1     Running     1          12h
openshift-cluster-machine-approver           machine-approver-f99ddd99f-bst6c                         1/1     Running     0          12h
openshift-cluster-version                    cluster-version-operator-865b5b8d46-gf2j4                1/1     Running     1          12h
openshift-controller-manager-operator        openshift-controller-manager-operator-58f688fbd9-kvxg2   1/1     Running     2          12h
openshift-controller-manager                 controller-manager-fhzxf                                 1/1     Running     35         12h
openshift-controller-manager                 controller-manager-r7xtx                                 1/1     Running     40         12h
openshift-controller-manager                 controller-manager-wrf2b                                 1/1     Running     49         12h
openshift-dns-operator                       dns-operator-d864b4d65-5g68h                             1/1     Running     0          12h
openshift-dns                                dns-default-j7n9t                                        2/2     Running     0          11h
openshift-dns                                dns-default-krv6q                                        2/2     Running     0          12h
openshift-dns                                dns-default-m6k72                                        2/2     Running     0          11h
openshift-dns                                dns-default-sgxs7                                        2/2     Running     0          12h
openshift-dns                                dns-default-vl9t7                                        2/2     Running     0          12h
openshift-dns                                dns-default-w65sk                                        2/2     Running     0          11h
openshift-etcd                               etcd-member-ostest-wg9bj-master-0                        2/2     Running     0          12h
openshift-etcd                               etcd-member-ostest-wg9bj-master-1                        2/2     Running     0          12h
openshift-etcd                               etcd-member-ostest-wg9bj-master-2                        2/2     Running     0          12h
openshift-insights                           insights-operator-5568799bf7-wzpv6                       1/1     Running     0          12h
openshift-kube-apiserver-operator            kube-apiserver-operator-dc675f755-h5drq                  1/1     Running     2          12h
openshift-kube-apiserver                     installer-2-ostest-wg9bj-master-0                        0/1     Completed   0          11h
openshift-kube-apiserver                     installer-2-ostest-wg9bj-master-1                        0/1     Completed   0          12h
openshift-kube-apiserver                     installer-2-ostest-wg9bj-master-2                        0/1     Completed   0          12h
openshift-kube-apiserver                     installer-3-ostest-wg9bj-master-0                        0/1     Completed   0          11h
openshift-kube-apiserver                     installer-3-ostest-wg9bj-master-1                        0/1     Completed   0          11h
openshift-kube-apiserver                     installer-3-ostest-wg9bj-master-2                        0/1     Completed   0          11h
openshift-kube-apiserver                     kube-apiserver-ostest-wg9bj-master-0                     3/3     Running     0          11h
openshift-kube-apiserver                     kube-apiserver-ostest-wg9bj-master-1                     3/3     Running     0          11h
openshift-kube-apiserver                     kube-apiserver-ostest-wg9bj-master-2                     3/3     Running     0          11h
openshift-kube-apiserver                     revision-pruner-2-ostest-wg9bj-master-0                  0/1     Completed   0          11h
openshift-kube-apiserver                     revision-pruner-2-ostest-wg9bj-master-1                  0/1     Completed   0          11h
openshift-kube-apiserver                     revision-pruner-2-ostest-wg9bj-master-2                  0/1     Completed   0          11h
openshift-kube-apiserver                     revision-pruner-3-ostest-wg9bj-master-0                  0/1     Completed   0          11h
openshift-kube-apiserver                     revision-pruner-3-ostest-wg9bj-master-1                  0/1     Completed   0          11h
openshift-kube-apiserver                     revision-pruner-3-ostest-wg9bj-master-2                  0/1     Completed   0          11h
openshift-kube-controller-manager-operator   kube-controller-manager-operator-586f864f89-crr88        1/1     Running     2          12h
openshift-kube-controller-manager            installer-2-ostest-wg9bj-master-0                        0/1     Completed   0          12h
openshift-kube-controller-manager            installer-6-ostest-wg9bj-master-0                        0/1     Completed   0          11h
openshift-kube-controller-manager            installer-6-ostest-wg9bj-master-1                        0/1     Completed   0          12h
openshift-kube-controller-manager            installer-6-ostest-wg9bj-master-2                        0/1     Completed   0          11h
openshift-kube-controller-manager            installer-7-ostest-wg9bj-master-0                        0/1     Completed   0          11h
openshift-kube-controller-manager            installer-7-ostest-wg9bj-master-1                        0/1     Completed   0          11h
openshift-kube-controller-manager            installer-7-ostest-wg9bj-master-2                        0/1     Completed   0          11h
openshift-kube-controller-manager            kube-controller-manager-ostest-wg9bj-master-0            2/2     Running     1          11h
openshift-kube-controller-manager            kube-controller-manager-ostest-wg9bj-master-1            2/2     Running     0          11h
openshift-kube-controller-manager            kube-controller-manager-ostest-wg9bj-master-2            2/2     Running     1          11h
openshift-kube-controller-manager            revision-pruner-2-ostest-wg9bj-master-0                  0/1     Completed   0          12h
openshift-kube-controller-manager            revision-pruner-6-ostest-wg9bj-master-0                  0/1     Completed   0          11h
openshift-kube-controller-manager            revision-pruner-6-ostest-wg9bj-master-1                  0/1     Completed   0          11h
openshift-kube-controller-manager            revision-pruner-6-ostest-wg9bj-master-2                  0/1     Completed   0          11h
openshift-kube-controller-manager            revision-pruner-7-ostest-wg9bj-master-0                  0/1     Completed   0          11h
openshift-kube-controller-manager            revision-pruner-7-ostest-wg9bj-master-1                  0/1     Completed   0          11h
openshift-kube-controller-manager            revision-pruner-7-ostest-wg9bj-master-2                  0/1     Completed   0          11h
openshift-kube-scheduler-operator            openshift-kube-scheduler-operator-644d57c785-wqtxg       1/1     Running     2          12h
openshift-kube-scheduler                     installer-2-ostest-wg9bj-master-2                        0/1     Completed   0          12h
openshift-kube-scheduler                     installer-4-ostest-wg9bj-master-0                        0/1     Completed   0          12h
openshift-kube-scheduler                     installer-4-ostest-wg9bj-master-1                        0/1     Completed   0          11h
openshift-kube-scheduler                     installer-4-ostest-wg9bj-master-2                        0/1     Completed   0          11h
openshift-kube-scheduler                     openshift-kube-scheduler-ostest-wg9bj-master-0           1/1     Running     0          12h
openshift-kube-scheduler                     openshift-kube-scheduler-ostest-wg9bj-master-1           1/1     Running     1          11h
openshift-kube-scheduler                     openshift-kube-scheduler-ostest-wg9bj-master-2           1/1     Running     1          11h
openshift-kube-scheduler                     revision-pruner-2-ostest-wg9bj-master-2                  0/1     Completed   0          12h
openshift-kube-scheduler                     revision-pruner-4-ostest-wg9bj-master-0                  0/1     Completed   0          11h
openshift-kube-scheduler                     revision-pruner-4-ostest-wg9bj-master-1                  0/1     Completed   0          11h
openshift-kube-scheduler                     revision-pruner-4-ostest-wg9bj-master-2                  0/1     Completed   0          11h
openshift-kuryr                              kuryr-cni-48f6d                                          1/1     Running     0          12h
openshift-kuryr                              kuryr-cni-55hw8                                          1/1     Running     0          12h
openshift-kuryr                              kuryr-cni-6swr7                                          1/1     Running     0          12h
openshift-kuryr                              kuryr-cni-dqdc7                                          1/1     Running     0          11h
openshift-kuryr                              kuryr-cni-gszvd                                          1/1     Running     0          11h
openshift-kuryr                              kuryr-cni-z72wc                                          1/1     Running     0          11h
openshift-kuryr                              kuryr-controller-8557787669-ldqj5                        1/1     Running     3          12h
openshift-kuryr                              kuryr-dns-admission-controller-8wx5l                     1/1     Running     0          12h
openshift-kuryr                              kuryr-dns-admission-controller-lqdq6                     1/1     Running     0          12h
openshift-kuryr                              kuryr-dns-admission-controller-z2zbp                     1/1     Running     0          12h
openshift-machine-api                        machine-api-controllers-7b9c567969-rw52l                 3/3     Running     0          12h
openshift-machine-api                        machine-api-operator-668d945596-j2xvc                    1/1     Running     1          12h
openshift-machine-config-operator            etcd-quorum-guard-74fbfccd87-2mrgb                       1/1     Running     0          12h
openshift-machine-config-operator            etcd-quorum-guard-74fbfccd87-7qgkk                       1/1     Running     0          12h
openshift-machine-config-operator            etcd-quorum-guard-74fbfccd87-hmtqn                       1/1     Running     0          12h
openshift-machine-config-operator            machine-config-controller-c97d65bd7-9zt4t                1/1     Running     1          12h
openshift-machine-config-operator            machine-config-daemon-c648v                              1/1     Running     0          11h
openshift-machine-config-operator            machine-config-daemon-hrcn6                              1/1     Running     0          12h
openshift-machine-config-operator            machine-config-daemon-kkgft                              1/1     Running     0          11h
openshift-machine-config-operator            machine-config-daemon-ng22z                              1/1     Running     0          11h
openshift-machine-config-operator            machine-config-daemon-nwjs6                              1/1     Running     0          12h
openshift-machine-config-operator            machine-config-daemon-sdpsp                              1/1     Running     0          12h
openshift-machine-config-operator            machine-config-operator-8467996985-w7skh                 1/1     Running     1          12h
openshift-machine-config-operator            machine-config-server-89d5w                              1/1     Running     0          12h
openshift-machine-config-operator            machine-config-server-cbhhv                              1/1     Running     0          12h
openshift-machine-config-operator            machine-config-server-xjxfx                              1/1     Running     0          12h
openshift-multus                             multus-2qfng                                             1/1     Running     0          11h
openshift-multus                             multus-8zgpl                                             1/1     Running     0          11h
openshift-multus                             multus-admission-controller-2zbgt                        1/1     Running     0          12h
openshift-multus                             multus-admission-controller-94jtk                        1/1     Running     0          12h
openshift-multus                             multus-admission-controller-z5pjg                        1/1     Running     0          12h
openshift-multus                             multus-h7h27                                             1/1     Running     0          12h
openshift-multus                             multus-hf87d                                             1/1     Running     0          12h
openshift-multus                             multus-nc9vz                                             1/1     Running     0          12h
openshift-multus                             multus-z5sgz                                             1/1     Running     0          11h
openshift-network-operator                   network-operator-7fdc49495d-44nfj                        1/1     Running     0          12h
openshift-openstack-infra                    coredns-ostest-wg9bj-master-0                            1/1     Running     0          12h
openshift-openstack-infra                    coredns-ostest-wg9bj-master-1                            1/1     Running     0          12h
openshift-openstack-infra                    coredns-ostest-wg9bj-master-2                            1/1     Running     0          12h
openshift-openstack-infra                    coredns-ostest-wg9bj-worker-cmnlk                        1/1     Running     0          11h
openshift-openstack-infra                    coredns-ostest-wg9bj-worker-r4sj5                        1/1     Running     0          11h
openshift-openstack-infra                    coredns-ostest-wg9bj-worker-wdmxw                        1/1     Running     0          11h
openshift-openstack-infra                    haproxy-ostest-wg9bj-master-0                            2/2     Running     1          12h
openshift-openstack-infra                    haproxy-ostest-wg9bj-master-1                            2/2     Running     1          12h
openshift-openstack-infra                    haproxy-ostest-wg9bj-master-2                            2/2     Running     1          12h
openshift-openstack-infra                    keepalived-ostest-wg9bj-master-0                         1/1     Running     0          12h
openshift-openstack-infra                    keepalived-ostest-wg9bj-master-1                         1/1     Running     0          12h
openshift-openstack-infra                    keepalived-ostest-wg9bj-master-2                         1/1     Running     0          12h
openshift-openstack-infra                    keepalived-ostest-wg9bj-worker-cmnlk                     1/1     Running     0          11h
openshift-openstack-infra                    keepalived-ostest-wg9bj-worker-r4sj5                     1/1     Running     0          11h
openshift-openstack-infra                    keepalived-ostest-wg9bj-worker-wdmxw                     1/1     Running     0          11h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-master-0                     1/1     Running     0          12h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-master-1                     1/1     Running     0          12h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-master-2                     1/1     Running     0          12h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-worker-cmnlk                 1/1     Running     0          11h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-worker-r4sj5                 1/1     Running     0          11h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-worker-wdmxw                 1/1     Running     0          11h
openshift-operator-lifecycle-manager         catalog-operator-67d65c7774-hk7pk                        1/1     Running     0          12h
openshift-operator-lifecycle-manager         olm-operator-8559ddbb5c-5dxv2                            1/1     Running     0          12h
openshift-operator-lifecycle-manager         packageserver-b5689448f-wdd4v                            1/1     Running     0          11h
openshift-operator-lifecycle-manager         packageserver-b5689448f-whhzk                            1/1     Running     0          11h
openshift-service-ca-operator                service-ca-operator-8655757d55-jz5h6                     1/1     Running     1          12h
openshift-service-ca                         apiservice-cabundle-injector-7c65c88bd9-fpm7x            1/1     Running     1          12h
openshift-service-ca                         configmap-cabundle-injector-6864bf47f7-9pfv2             1/1     Running     1          12h
openshift-service-ca                         service-serving-cert-signer-77c9c666c9-gtbsl             1/1     Running     1          12h






(shiftstack) [stack@undercloud-0 ~]$ openstack port list | grep 172.30.0.1
| 0abd15ce-2536-452a-b2b9-054421f506d5 | octavia-lb-dbbae311-14c3-4298-a228-4df5e500275e      | fa:16:3e:43:73:11 | ip_address='172.30.0.10', subnet_id='d4c54fd6-9550-4eee-b590-a46abe5a721f'    | DOWN   |
| 57d29893-fbb9-4b2a-93e0-6b07ab5dc8e7 | octavia-lb-9f6f2c5f-4fdd-4986-8d39-1d59768206ef      | fa:16:3e:2b:58:a8 | ip_address='172.30.0.1', subnet_id='d4c54fd6-9550-4eee-b590-a46abe5a721f'     | DOWN   |

Comment 1 Ben Bennett 2019-12-10 14:20:50 UTC
Pushing to 4.4 since this is not a 4.3 regression.  When we have a fix for it, we can talk about doing a back-port.

Comment 2 Luis Tomas Bolivar 2019-12-17 15:49:25 UTC
This turns out to be a bug in neutron when several subnets/networks are created and connected to the same router around the same time.

After investigation, the problem is that even though the created subnet is connected to the router, it is not properly configured:
# Subnet
$ openstack subnet list | grep 10.128.36 
| 6926d902-999e-4faf-94f4-7b2c6ca35a14 | ns/openshift-kube-scheduler-operator-subnet                     | 54f03a0a-e6e5-43d4-912d-3996d5376da6 | 10.128.36.0/23 |

# It even has an ACTIVE gw port (10.128.36.1)
$ openstack port list | grep 10.128.36.
| 6242053e-fd7c-4a5a-8acf-bae6325db33d |                                                      | fa:16:3e:9e:a4:26 | ip_address='10.128.36.7', subnet_id='6926d902-999e-4faf-94f4-7b2c6ca35a14'    | ACTIVE |
| 68ec4c36-e7aa-44f6-b250-fe7169e70bb4 |                                                      | fa:16:3e:79:ab:75 | ip_address='10.128.36.9', subnet_id='6926d902-999e-4faf-94f4-7b2c6ca35a14'    | ACTIVE |
| 79ee6f2c-8c27-4bda-a7be-918911482fd7 |                                                      | fa:16:3e:e8:9c:d7 | ip_address='10.128.36.1', subnet_id='6926d902-999e-4faf-94f4-7b2c6ca35a14'    | ACTIVE |
| e868089c-49e7-448c-af9b-e565d2bdd170 |                                                      | fa:16:3e:2d:15:30 | ip_address='10.128.36.5', subnet_id='6926d902-999e-4faf-94f4-7b2c6ca35a14'    | ACTIVE |

# And the router shows it is connected to it
$ openstack router show ostest-rnf5x-external-router | grep 6926d902-999e-4faf-94f4-7b2c6ca35a14                                                                                 | interfaces_info         | [{"subnet_id": "4d21e651-ae15-4196-ac20-35235f641afe", "ip_address": "10.128.12.1", "port_id": "02192a04-3fec-4ea5-b77e-4285135c683a"}, {"subnet_id": "3fb1a51a-01ab-41cf-a036-adbe7f98aafa", "ip_address": "10.128.56.1", "port_id": "03d98330-5b59-4c8c-b75d-7c5c570fa50a"}, {"subnet_id": "10a8c71c-9b00-4fc3-84ca-341279ec35c7", "ip_address": "10.128.0.1", "port_id": "0619ade6-0528-4689-8dd6-b0871e7e17b6"}, {"subnet_id": "02035a8d-b77c-49c4-8e28-4f9f515c52bc", "ip_address": "10.128.48.1", "port_id": "0a3ec172-d3d3-4d51-84ae-28e0acfa4a43"}, {"subnet_id": "3557cc91-8eaf-4eb1-9484-0e69abe34175", "ip_address": "10.128.20.1", "port_id": "11587773-fbc4-4cca-8869-8f47354fba99"}, {"subnet_id": "7d445abd-98e3-4c3c-8d74-4fb5161172f1", "ip_address": "10.128.84.1", "port_id": "11ca52f8-e4ba-49d2-aa5f-c8d6f7a1e65a"}, {"subnet_id": "68c03029-2ba4-41f2-bd1b-fed8a1d971f9", "ip_address": "10.128.50.1", "port_id": "1547deca-4a4c-465c-b3d6-62c419684c61"}, {"subnet_id": "1394e64c-f865-41b3-872c-bc559ea9d128", "ip_address": "10.128.74.1", "port_id": "1f153eff-f024-4da9-aeb2-8f388747eade"}, {"subnet_id": "da19149d-3f28-45cc-ae1b-4a006dd05797", "ip_address": "10.128.72.1", "port_id": "207c52ad-516c-417f-8f6c-37916856ece0"}, {"subnet_id": "62247aec-42c0-4565-ab77-abf3c4319bfb", "ip_address": "10.128.52.1", "port_id": "20996db4-e16b-479c-b437-6f0eb985b322"}, {"subnet_id": "8ad5c87b-be8c-469a-8610-37315d85a363", "ip_address": "10.128.10.1", "port_id": "20f08e2c-f1a8-436b-8380-32f45616d61b"}, {"subnet_id": "eb56f46d-577b-4da4-9e57-33f2a4e0c1cf", "ip_address": "10.128.42.1", "port_id": "27de3cc3-cd6f-4974-aa71-9c0f975dff03"}, {"subnet_id": "e1de8d8a-8ec7-43d4-a94b-6e68ec4564e5", "ip_address": "10.128.70.1", "port_id": "2b704b41-7a6b-4c64-93cf-409575907f2e"}, {"subnet_id": "d84d916a-cad1-4c60-96a2-5d9b954d6ea5", "ip_address": "10.128.30.1", "port_id": "30746d92-bde9-47dc-a546-dc62406ee7f8"}, {"subnet_id": "bdefbeb1-26ea-4711-b511-fc661ef6b9df", "ip_address": "10.128.8.1", "port_id": "32f0f949-d23a-421a-9d70-82c17fc3f429"}, {"subnet_id": "9d8b14aa-7ec5-4ce7-b6ba-53a96fb0f26e", "ip_address": "10.128.28.1", "port_id": "3732f0ef-9cce-4149-8681-afee83a3b245"}, {"subnet_id": "1c0b2865-7c15-4f1f-acd0-acfaf7c44f62", "ip_address": "10.128.46.1", "port_id": "3bd7154f-836f-4cac-bb38-bf3a3c6ff7fe"}, {"subnet_id": "72861c77-5e9c-4c90-9411-b2957f5074ba", "ip_address": "10.128.86.1", "port_id": "3d2ab07c-4675-4caa-b2f8-2c918998db5b"}, {"subnet_id": "3205c5f1-e951-46e6-9b68-e130a3415744", "ip_address": "10.128.14.1", "port_id": "530b8fba-c84e-4841-bb73-3ea7a2ee107b"}, {"subnet_id": "1bd0328a-9dc7-4f09-adfb-542ec43b97f0", "ip_address": "10.128.58.1", "port_id": "5c5e5162-6484-4e5d-93c5-a9029e019fbc"}, {"subnet_id": "6d324273-0e0a-4d19-b528-5c3a3a246ed4", "ip_address": "10.128.92.1", "port_id": "5c7ea132-823b-416c-baab-62766c37b103"}, {"subnet_id": "b7d6fbfd-3416-4f54-a123-4295bdbbcda2", "ip_address": "10.128.80.1", "port_id": "616fdad5-997b-42e2-8e08-7e3ce06cc893"}, {"subnet_id": "c1874895-ad56-46fb-83a8-06783f6191ee", "ip_address": "10.128.38.1", "port_id": "7421cd39-9e3e-4b7e-945c-f979442ead58"}, {"subnet_id": "32464597-9d13-4d8c-a288-9d6e008cc49d", "ip_address": "10.128.82.1", "port_id": "79c05d72-4691-4104-b185-ba28d2f1b9af"}, {"subnet_id": "6926d902-999e-4faf-94f4-7b2c6ca35a14", "ip_address": "10.128.36.1", "port_id": "79ee6f2c-8c27-4bda-a7be-918911482fd7"}, {"subnet_id": "996b3c6d-322c-4314-901f-f2d17841f486", "ip_address": "10.128.32.1", "port_id": "7fed9645-ea8b-404e-94a6-9d31a316c3d4"}, {"subnet_id": "94de1b87-34bb-459c-af08-e4c6b9e0476c", "ip_address": "10.128.90.1", "port_id": "85dede0c-b4b0-4d8a-a188-7820b91ecc20"}, {"subnet_id": "738217f5-55f0-4b23-8e3e-cbca72db1a01", "ip_address": "10.196.0.1", "port_id": "87a6b915-4d9b-4efb-82a1-e4fd999f4f45"}, {"subnet_id": "e36ce6e9-b6c8-44ec-ae7b-7c03566b9b91", "ip_address": "10.128.66.1", "port_id": "882be296-5402-44bd-af18-92ec0e063cde"}, {"subnet_id": "6924e610-b2e3-47f1-b768-76e51f4b94f0", "ip_address": "10.128.54.1", "port_id": "93b65d38-df88-4fb1-b3fb-c6fe42c053be"}, {"subnet_id": "5ac2915a-58c5-47db-9be5-21d23da036ab", "ip_address": "10.128.4.1", "port_id": "98b01ba1-8ae2-41f2-b541-49515d851fdb"}, {"subnet_id": "ff7b620f-a519-45da-8758-d6756d405dfe", "ip_address": "10.128.64.1", "port_id": "9913d8aa-a15b-4216-ac46-b23162c14503"}, {"subnet_id": "c2d91acf-4548-48c5-9813-2529611d1d2c", "ip_address": "10.128.34.1", "port_id": "9cb40889-76b7-4580-bbbf-8d9d6da6700a"}, {"subnet_id": "09170430-8dfb-4622-a4d2-5b91400f5e51", "ip_address": "10.128.88.1", "port_id": "9edca8cc-58dd-49be-b563-f45ca06b9ba0"}, {"subnet_id": "bb441960-4436-4eb9-b571-eddbb3364dab", "ip_address": "172.31.255.254", "port_id": "9ff837c6-8239-45b3-a41e-8d784a8c237d"}, {"subnet_id": "b197dc72-4f4f-484d-ab1f-12b96cdef52d", "ip_address": "10.128.78.1", "port_id": "a436a35b-e23e-4a65-b320-b0f2c1e29bb0"}, {"subnet_id": "5f79b732-ff7a-4b1a-bdfa-a02808e0307e", "ip_address": "10.128.6.1", "port_id": "ad5d1f96-3f44-4e8e-820f-f3509abeb33b"}, {"subnet_id": "bb42e126-a712-4848-934d-23caef0c6b98", "ip_address": "10.128.76.1", "port_id": "b5bdbd16-eaa4-4a0b-a6c4-cd62453e62e3"}, {"subnet_id": "71a9fd17-e8d1-464f-9b56-e6c508f68406", "ip_address": "10.128.62.1", "port_id": "b6896c37-b914-40fe-9706-37ffdcf3888e"}, {"subnet_id": "0d028514-77a7-42c1-90bb-6896eead2775", "ip_address": "10.128.22.1", "port_id": "c3c6a267-e9f2-4dbd-aa47-fbbf0f0fe655"}, {"subnet_id": "4f65263f-b11f-4d94-81f5-bfc46217e08b", "ip_address": "10.128.68.1", "port_id": "c6c7f19b-870c-4d3e-b9fa-240b12468a11"}, {"subnet_id": "db6529c5-c708-47ef-8116-15c402dfb5fd", "ip_address": "10.128.60.1", "port_id": "d74e3d49-a849-48fe-a24c-2b28e05f563b"}, {"subnet_id": "d988cbda-55f3-45f6-8460-9f3e4b4180e7", "ip_address": "10.128.2.1", "port_id": "d84a8ffc-547c-432b-b24f-b776d92446ac"}, {"subnet_id": "0947c7c9-4ff1-47c9-815e-933294c2b823", "ip_address": "10.128.26.1", "port_id": "d8d89031-445b-4961-9cb9-f547923843cb"}, {"subnet_id": "f1dd9ebe-764f-4df6-b8e1-ffc485058437", "ip_address": "10.128.16.1", "port_id": "dd4802c8-bd4e-4952-b3da-79bc8b4e6527"}, {"subnet_id": "1598b795-e21d-487f-811b-c518cbfbe760", "ip_address": "10.128.24.1", "port_id": "df64ae76-42c5-4a2c-8a42-bd4d75a01108"}, {"subnet_id": "7b1ddde7-0261-41a0-af57-89a0747938a1", "ip_address": "10.128.18.1", "port_id": "fab510a2-1dcc-44cc-9ccd-3883302a7bd2"}, {"subnet_id": "9d129f83-d623-4bd9-bf7f-f20564ae3d08", "ip_address": "10.128.44.1", "port_id": "fec72e95-0fe9-4d51-9fbc-08086f4fab23"}] |


The ports in this subnet don't have access to the gateway router. I've check the ARP request is reaching the qrouter namespace in the controller, but there is no ARP response. After checking the qr-XXX configuration related to the subnet, it turns out it is missing the IP:

302: qr-79ee6f2c-8c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:e8:9c:d7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f816:3eff:fee8:9cd7/64 scope link 
       valid_lft forever preferred_lft forever


And after manually adding it (or by detaching and attaching again the subnet to/from the router), the connectivity starts working:
302: qr-79ee6f2c-8c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:e8:9c:d7 brd ff:ff:ff:ff:ff:ff
    inet 10.128.36.1/24 scope global qr-79ee6f2c-8c
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fee8:9cd7/64 scope link 
       valid_lft forever preferred_lft forever

Comment 3 Luis Tomas Bolivar 2019-12-17 16:15:17 UTC
*** Bug 1783258 has been marked as a duplicate of this bug. ***

Comment 5 Slawek Kaplonski 2019-12-19 09:44:28 UTC
I was debugging it on neutron's side last 2 days. I added some additional debug logs in test env and here is what I found so far.

Port which was failed to be configured:

+-----------------------+-----------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                     |
+-----------------------+-----------------------------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                                      |
| allowed_address_pairs |                                                                                                           |
| binding:vnic_type     | normal                                                                                                    |
| created_at            | 2019-12-19T08:28:02Z                                                                                      |
| description           |                                                                                                           |
| device_id             | 60df726a-95d9-4f72-9461-b830c114aa0e                                                                      |
| device_owner          | network:ha_router_replicated_interface                                                                    |
| dns_assignment        | {"hostname": "host-10-128-10-1", "ip_address": "10.128.10.1", "fqdn": "host-10-128-10-1.shiftstack.com."} |
| dns_domain            |                                                                                                           |
| dns_name              |                                                                                                           |
| extra_dhcp_opts       |                                                                                                           |
| fixed_ips             | {"subnet_id": "b73f874d-faa5-4d0d-8fee-063994ee66e8", "ip_address": "10.128.10.1"}                        |
| id                    | f448ad14-8d0a-4912-b787-aa5e86b8c357                                                                      |
| mac_address           | fa:16:3e:7f:b3:fa                                                                                         |
| name                  |                                                                                                           |
| network_id            | 256941d3-412a-4822-b5e7-3d9a74b38ffd                                                                      |
| port_security_enabled | False                                                                                                     |
| project_id            | e8b0db94fa684f9c93377c9ee7774677                                                                          |
| qos_policy_id         |                                                                                                           |
| revision_number       | 12                                                                                                        |
| security_groups       |                                                                                                           |
| status                | ACTIVE                                                                                                    |
| tags                  |                                                                                                           |
| tenant_id             | e8b0db94fa684f9c93377c9ee7774677                                                                          |
| updated_at            | 2019-12-19T08:28:53Z                                                                                      |
+-----------------------+-----------------------------------------------------------------------------------------------------------+

It's IP address is 10.128.10.1 and it's from subnet:

+-------------------+--------------------------------------------------+
| Field             | Value                                            |
+-------------------+--------------------------------------------------+
| allocation_pools  | {"start": "10.128.10.2", "end": "10.128.11.254"} |
| cidr              | 10.128.10.0/23                                   |
| created_at        | 2019-12-19T08:27:55Z                             |
| description       |                                                  |
| dns_nameservers   |                                                  |
| enable_dhcp       | False                                            |
| gateway_ip        | 10.128.10.1                                      |
| host_routes       |                                                  |
| id                | b73f874d-faa5-4d0d-8fee-063994ee66e8             |
| ip_version        | 4                                                |
| ipv6_address_mode |                                                  |
| ipv6_ra_mode      |                                                  |
| name              | ns/openshift-service-ca-operator-subnet          |
| network_id        | 256941d3-412a-4822-b5e7-3d9a74b38ffd             |
| project_id        | e8b0db94fa684f9c93377c9ee7774677                 |
| revision_number   | 1                                                |
| service_types     |                                                  |
| subnetpool_id     | 9eb85e53-33d5-4cd2-be60-09df12f7db34             |
| tags              | openshiftClusterID=ostest-nxqct                  |
| tenant_id         | e8b0db94fa684f9c93377c9ee7774677                 |
| updated_at        | 2019-12-19T08:27:59Z                             |
+-------------------+--------------------------------------------------+

Now, lets check neutron L3 agent logs.
This port was configured and added to Keepalived's VIPs here:

2019-12-19 08:28:22.632 223011 DEBUG neutron.agent.linux.keepalived [-] SK: adding VIP [fe80::f816:3eff:fe7f:b3fa/64, qr-f448ad14-8d, link] add_vip /usr/lib/python2.7/site-packages/neutron/agent/linux/keepalived.py:20

but L3 agent said that same vip is already added in the vips list:

2019-12-19 08:28:22.633 223011 DEBUG neutron.agent.linux.keepalived [-] VIP [10.128.10.1/23, qr-f448ad14-8d, None] already present in [...]

This list contains instances of class KeepalivedVipAddress which is defined in https://github.com/openstack/neutron/blob/stable/queens/neutron/agent/linux/keepalived.py#L85
Same element is element with same IP address. So, it seems that we have already added 10.128.10.1 in vips list. Quick look at this list and we have:

[10.128.10.1/23, qr-f3c06bfa-dd, None]

So, lets now look for port which starts with f3c06bfa-dd.
There is no such port in neutron db anymore:

(shiftstack) [stack@undercloud-0 ~]$ neutron port-list | grep f3c06bfa-dd
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.

Next step is to look at L3 agent logs again.
There we can see that such port was added about 1 minute before our broken port:

2019-12-19 08:27:39.856 223011 DEBUG neutron.agent.l3.router_info [-] appending port {u'allowed_address_pairs': [], u'extra_dhcp_opts': [], u'updated_at': u'2019-12-19T08:27:23Z', u'dns_domain': u'', u'device_owner': u'network:ha_router_replicated_interface', u'revision_number': 7, u'port_security_enabled': False, u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'7a9f9c7c-da57-4666-af5d-de0fd3840a41', u'prefixlen': 23, u'ip_address': u'10.128.10.1'}], u'id': u'f3c06bfa-ddc7-4e4e-a294-7fd03b076531', u'security_groups': [], u'binding:vif_details': {}, u'address_scopes': {u'4': None, u'6': None}, u'binding:vif_type': u'unbound', u'qos_policy_id': None, u'mac_address': u'fa:16:3e:22:ad:ab', u'project_id': u'e8b0db94fa684f9c93377c9ee7774677', u'status': u'DOWN', u'subnets': [{u'dns_nameservers': [], u'ipv6_ra_mode': None, u'gateway_ip': u'10.128.10.1', u'cidr': u'10.128.10.0/23', u'id': u'7a9f9c7c-da57-4666-af5d-de0fd3840a41', u'subnetpool_id': u'9eb85e53-33d5-4cd2-be60-09df12f7db34'}], u'binding:host_id': u'', u'description': u'', u'tags': [], u'dns_assignment': [{u'hostname': u'host-10-128-10-1', u'ip_address': u'10.128.10.1', u'fqdn': u'host-10-128-10-1.shiftstack.com.'}], u'device_id': u'60df726a-95d9-4f72-9461-b830c114aa0e', u'name': u'', u'admin_state_up': True, u'network_id': u'd4f8c742-7553-43bc-aaf8-d167b74a02da', u'dns_name': u'', u'created_at': u'2019-12-19T08:27:20Z', u'mtu': 1450, u'tenant_id': u'e8b0db94fa684f9c93377c9ee7774677', u'extra_subnets': []} to internal_ports cache _process_internal_ports /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:57

and it was in subnet a9f9c7c-da57-4666-af5d-de0fd3840a41 which also don't exists anymore.

Next look at L3 agent gives us the answer. This port and this subnet was removed from L3 agent on:

2019-12-19 08:28:23.383 223011 DEBUG neutron.agent.l3.router_info [-] removing port {u'allowed_address_pairs': [], u'extra_dhcp_opts': [], u'updated_at': u'2019-12-19T08:27:23Z', u'dns_domain': u'', u'device_owner': u'network:ha_router_replicated_interface', u'revision_number': 7, u'port_security_enabled': False, u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'7a9f9c7c-da57-4666-af5d-de0fd3840a41', u'prefixlen': 23, u'ip_address': u'10.128.10.1'}], u'id': u'f3c06bfa-ddc7-4e4e-a294-7fd03b076531', u'security_groups': [], u'binding:vif_details': {}, u'address_scopes': {u'4': None, u'6': None}, u'binding:vif_type': u'unbound', u'qos_policy_id': None, u'mac_address': u'fa:16:3e:22:ad:ab', u'project_id': u'e8b0db94fa684f9c93377c9ee7774677', u'status': u'DOWN', u'subnets': [{u'dns_nameservers': [], u'ipv6_ra_mode': None, u'gateway_ip': u'10.128.10.1', u'cidr': u'10.128.10.0/23', u'id': u'7a9f9c7c-da57-4666-af5dde0fd3840a41', u'subnetpool_id': u'9eb85e53-33d5-4cd2-be60-09df12f7db34'}], u'binding:host_id': u'', u'description': u'', u'tags': [], u'dns_assignment': [{u'hostname': u'host-10-128-10-1', u'ip_address': u'10.128.10.1', u'fqdn': u'host10-128-10-1.shiftstack.com.'}], u'device_id': u'60df726a-95d9-4f72-9461-b830c114aa0e', u'name': u'', u'admin_state_up': True, u'network_id': u'd4f8c742-7553-43bc-aaf8-d167b74a02da', u'dns_name': u'', u'created_at': u'2019-12-19T08:27:20Z', u'mtu': 1450, u'tenant_id': u'e8b0db94fa684f9c93377c9ee7774677', u'extra_subnets': []} from internal_ports cache _process_internal_ports /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:591


So to sum up, issue was caused because first port with 10.128.10.1 was added at 8:27:39 on port  f3c06bfa-ddc7-4e4e-a294-7fd03b076531 and after that it was trying to add same IP address on different port 'f448ad14-8d0a-4912-b787-aa5e86b8c357' at 8:28:16. After that at 8:28:23 port was removed f3c06bfa-ddc7-4e4e-a294-7fd03b076531 but our IP address wasn't configured anymore on existing port.

Comment 11 Alex Katz 2020-02-25 11:37:43 UTC
The fix is verified on OpenStack-13.0-RHEL-7-20200214.1:

 1. Create several subnets from pool
 2. Attach subnets to router
 3. Detach subnets from router
 4. Delete subnets
 5. GOTO 1

Comment 13 errata-xmlrpc 2020-03-10 11:26:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0770


Note You need to log in before you can comment on or make changes to this bug.