Bug 1783258 - [IPI] [OSP] [Kuryr] On some HW, the installation fails due to the startup order of operators that caused by a delay in OSP components
Summary: [IPI] [OSP] [Kuryr] On some HW, the installation fails due to the startup ord...
Keywords:
Status: CLOSED DUPLICATE of bug 1779654
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.2.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.3.z
Assignee: Luis Tomas Bolivar
QA Contact: GenadiC
URL:
Whiteboard:
Depends On: 1779654 1790416 1790417
Blocks: 1751756
TreeView+ depends on / blocked
 
Reported: 2019-12-13 13:07 UTC by Luis Tomas Bolivar
Modified: 2020-01-13 10:44 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1779654
Environment:
Last Closed: 2019-12-17 16:15:17 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Luis Tomas Bolivar 2019-12-13 13:07:39 UTC
+++ This bug was initially created as a clone of Bug #1779654 +++

Description of problem:
On some HW, the installation fails due to the startup order of operators that caused by a delay in OSP components.
Debug info below

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP13
2. Install OCP 4.2 with Kuryr
3.

Actual results:
installation failed

Expected results:
Installation passed

Additional info:
(undercloud) [stack@undercloud-0 ~]$ oc get nodes
NAME                        STATUS   ROLES    AGE   VERSION
ostest-wg9bj-master-0       Ready    master   12h   v1.14.6+31a56cf75
ostest-wg9bj-master-1       Ready    master   12h   v1.14.6+31a56cf75
ostest-wg9bj-master-2       Ready    master   12h   v1.14.6+31a56cf75
ostest-wg9bj-worker-cmnlk   Ready    worker   11h   v1.14.6+31a56cf75
ostest-wg9bj-worker-r4sj5   Ready    worker   11h   v1.14.6+31a56cf75
ostest-wg9bj-worker-wdmxw   Ready    worker   11h   v1.14.6+31a56cf75
(undercloud) [stack@undercloud-0 ~]$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
cloud-credential                           4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
dns                                        4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
insights                                   4.2.0-0.nightly-2019-11-26-164052   True        False         True       11h
kube-apiserver                             4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
kube-controller-manager                    4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
kube-scheduler                             4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
machine-api                                4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
machine-config                             4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
network                                    4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
openshift-controller-manager               4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
operator-lifecycle-manager                 4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
operator-lifecycle-manager-catalog         4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
operator-lifecycle-manager-packageserver   4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
service-ca                                 4.2.0-0.nightly-2019-11-26-164052   True        False         False      11h
(undercloud) [stack@undercloud-0 ~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          12h     Unable to apply 4.2.0-0.nightly-2019-11-26-164052: an unknown error has occurred
(undercloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| 981b5fdd-54f8-42ce-83a4-439dcbb062aa | controller-1 | ACTIVE | ctlplane=192.168.24.20 | overcloud-full | controller |
| 3e092d44-4de0-49bf-b348-5f0cb5642378 | controller-2 | ACTIVE | ctlplane=192.168.24.17 | overcloud-full | controller |
| 694d4d76-45e1-4a4c-a02e-acd69d770b1d | compute-0    | ACTIVE | ctlplane=192.168.24.15 | overcloud-full | compute    |
| 6e62763e-1fbf-4254-9961-4416bec7d04a | controller-0 | ACTIVE | ctlplane=192.168.24.12 | overcloud-full | controller |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
(undercloud) [stack@undercloud-0 ~]$ . shiftstackrc
(shiftstack) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+---------------------------+--------+------------------------------------+-------+-----------+
| ID                                   | Name                      | Status | Networks                           | Image | Flavor    |
+--------------------------------------+---------------------------+--------+------------------------------------+-------+-----------+
| c343db2e-5941-4e0e-9b0a-c510031d7dce | ostest-wg9bj-worker-r4sj5 | ACTIVE | ostest-wg9bj-openshift=10.196.0.36 | rhcos | m4.xlarge |
| ae93ffae-db0c-4282-9a76-e5e1dca97e86 | ostest-wg9bj-worker-wdmxw | ACTIVE | ostest-wg9bj-openshift=10.196.0.38 | rhcos | m4.xlarge |
| af9784e1-e6f5-4f44-8058-b83df6c54d40 | ostest-wg9bj-worker-cmnlk | ACTIVE | ostest-wg9bj-openshift=10.196.0.34 | rhcos | m4.xlarge |
| f2a8fc24-78cc-44bf-95f4-a94775d15c63 | ostest-wg9bj-master-0     | ACTIVE | ostest-wg9bj-openshift=10.196.0.13 | rhcos | m4.xlarge |
| 719a23bc-85f4-42c1-aea2-a9afc62d95a4 | ostest-wg9bj-master-2     | ACTIVE | ostest-wg9bj-openshift=10.196.0.29 | rhcos | m4.xlarge |
| dd693a11-4fee-4149-a3c2-4c8717085f2d | ostest-wg9bj-master-1     | ACTIVE | ostest-wg9bj-openshift=10.196.0.15 | rhcos | m4.xlarge |
+--------------------------------------+---------------------------+--------+------------------------------------+-------+-----------+



(shiftstack) [stack@undercloud-0 ~]$ oc describe pod openshift-apiserver-operator-7648756bf6-snj67 -n openshift-apiserver-operator
Name:               openshift-apiserver-operator-7648756bf6-snj67
Namespace:          openshift-apiserver-operator
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               ostest-wg9bj-master-1/10.196.0.15
Start Time:         Tue, 26 Nov 2019 17:20:01 -0500
Labels:             app=openshift-apiserver-operator
                    pod-template-hash=7648756bf6
Annotations:        k8s.v1.cni.cncf.io/networks-status:
                      [{
                          "name": "kuryr",
                          "interface": "eth0",
                          "ips": [
                              "10.128.40.9"
                          ],
                          "mac": "fa:16:3e:b9:fd:e6",
                          "default": true,
                          "dns": {}
                      }]
                    openstack.org/kuryr-pod-label: {"app": "openshift-apiserver-operator", "pod-template-hash": "7648756bf6"}
                    openstack.org/kuryr-vif:
                      {"versioned_object.changes": ["default_vif"], "versioned_object.data": {"additional_vifs": {}, "default_vif": {"versioned_object.changes":...
Status:             Running
IP:                 10.128.40.9
Controlled By:      ReplicaSet/openshift-apiserver-operator-7648756bf6
Containers:
  openshift-apiserver-operator:
    Container ID:  cri-o://84e9889eda25003036998b2c0bbd5d2d06ca74f5f62a690a40b4601887f10f03
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7
    Image ID:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7
    Port:          8443/TCP
    Host Port:     0/TCP
    Command:
      cluster-openshift-apiserver-operator
      operator
    Args:
      --config=/var/run/configmaps/config/config.yaml
    State:       Waiting
      Reason:    CrashLoopBackOff
    Last State:  Terminated
      Reason:    Error
      Message:   I1127 10:19:30.592561       1 cmd.go:160] Using service-serving-cert provided certificates
I1127 10:19:30.593211       1 observer_polling.go:106] Starting file observer
W1127 10:19:33.664944       1 builder.go:181] unable to get owner reference (falling back to namespace): Get https://172.30.0.1:443/api/v1/namespaces/openshift-apiserver-operator/pods: dial tcp 172.30.0.1:443: connect: no route to host
F1127 10:20:04.555120       1 cmd.go:92] Get https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 172.30.0.1:443: connect: no route to host

      Exit Code:    255
      Started:      Wed, 27 Nov 2019 05:19:30 -0500
      Finished:     Wed, 27 Nov 2019 05:20:04 -0500
    Ready:          False
    Restart Count:  130
    Requests:
      cpu:     10m
      memory:  50Mi
    Environment:
      IMAGE:                   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:699ef43532027b2429eeb48b809e27c2c1b0e750b2e5fa0d84159540c141c7ca
      OPERATOR_IMAGE:          quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7
      OPERATOR_IMAGE_VERSION:  4.2.0-0.nightly-2019-11-26-164052
      OPERAND_IMAGE_VERSION:   4.2.0-0.nightly-2019-11-26-164052
    Mounts:
      /var/run/configmaps/config from config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from openshift-apiserver-operator-token-scnqv (ro)
      /var/run/secrets/serving-cert from serving-cert (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  serving-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  openshift-apiserver-operator-serving-cert
    Optional:    true
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openshift-apiserver-operator-config
    Optional:  false
  openshift-apiserver-operator-token-scnqv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  openshift-apiserver-operator-token-scnqv
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 120s
                 node.kubernetes.io/unreachable:NoExecute for 120s
Events:
  Type     Reason   Age                     From                            Message
  ----     ------   ----                    ----                            -------
  Normal   Pulled   24m (x126 over 11h)     kubelet, ostest-wg9bj-master-1  Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f04c416a210b19772d20b73d24cbe2da79f565f688eaef33029dbc2328fa4be7" already present on machine
  Warning  BackOff  4m19s (x2946 over 11h)  kubelet, ostest-wg9bj-master-1  Back-off restarting failed container
(shiftstack) [stack@undercloud-0 ~]$ #oc describe pod openshift-apiserver-operator-7648756bf6-snj67 -n openshift-apiserver-operator
(shiftstack) [stack@undercloud-0 ~]$ oc logs openshift-apiserver-operator-7648756bf6-snj67 -n openshift-apiserver-operator
I1127 10:19:30.592561       1 cmd.go:160] Using service-serving-cert provided certificates
I1127 10:19:30.593211       1 observer_polling.go:106] Starting file observer
W1127 10:19:33.664944       1 builder.go:181] unable to get owner reference (falling back to namespace): Get https://172.30.0.1:443/api/v1/namespaces/openshift-apiserver-operator/pods: dial tcp 172.30.0.1:443: connect: no route to host
F1127 10:20:04.555120       1 cmd.go:92] Get https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 172.30.0.1:443: connect: no route to host








(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer list
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+----------------+---------------------+----------+
| id                                   | name                                                                  | project_id                       | vip_address    | provisioning_status | provider |
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+----------------+---------------------+----------+
| 9f6f2c5f-4fdd-4986-8d39-1d59768206ef | ostest-wg9bj-kuryr-api-loadbalancer                                   | aa0378b6eb544d59a84c78a790f0a768 | 172.30.0.1     | ACTIVE              | octavia  |
| 07489b31-730e-4f2d-b7e8-83c8b6888ad2 | openshift-etcd/etcd                                                   | aa0378b6eb544d59a84c78a790f0a768 | 172.30.20.213  | ACTIVE              | octavia  |
| 28c0891a-addb-4d34-9221-83d2cd36b985 | openshift-cluster-version/cluster-version-operator                    | aa0378b6eb544d59a84c78a790f0a768 | 172.30.118.219 | ACTIVE              | octavia  |
| 9ce335fb-2565-447a-bdbb-a7be2d3f6769 | openshift-kube-scheduler-operator/metrics                             | aa0378b6eb544d59a84c78a790f0a768 | 172.30.130.117 | ACTIVE              | octavia  |
| 6ef04bb2-4378-43aa-b793-1f3dc2a177a6 | openshift-kube-controller-manager-operator/metrics                    | aa0378b6eb544d59a84c78a790f0a768 | 172.30.62.162  | ACTIVE              | octavia  |
| 1b1a1ffe-7048-457c-8e92-c8b53be2a30d | openshift-kuryr/kuryr-dns-admission-controller                        | aa0378b6eb544d59a84c78a790f0a768 | 172.30.62.33   | ACTIVE              | octavia  |
| 314956c5-e8f7-4505-8118-eb9c664a45d0 | openshift-machine-api/machine-api-operator                            | aa0378b6eb544d59a84c78a790f0a768 | 172.30.82.129  | ACTIVE              | octavia  |
| f35fa438-d40d-4454-ba00-046e74ad4b72 | openshift-apiserver-operator/metrics                                  | aa0378b6eb544d59a84c78a790f0a768 | 172.30.54.49   | ACTIVE              | octavia  |
| dfc5427c-89ca-4af1-b8c6-43d9ea6b8869 | openshift-cloud-credential-operator/controller-manager-service        | aa0378b6eb544d59a84c78a790f0a768 | 172.30.121.201 | ACTIVE              | octavia  |
| 46c4535d-ef50-473f-add5-1d7613639185 | openshift-kube-apiserver-operator/metrics                             | aa0378b6eb544d59a84c78a790f0a768 | 172.30.211.35  | ACTIVE              | octavia  |
| 854da7fc-d095-459e-a1b8-3e7de3a36bd0 | openshift-controller-manager-operator/metrics                         | aa0378b6eb544d59a84c78a790f0a768 | 172.30.56.92   | ACTIVE              | octavia  |
| dbbae311-14c3-4298-a228-4df5e500275e | openshift-dns/dns-default                                             | aa0378b6eb544d59a84c78a790f0a768 | 172.30.0.10    | ACTIVE              | octavia  |
| 74bd7e77-97ff-4e06-89c7-f6ed68e1a988 | openshift-multus/multus-admission-controller                          | aa0378b6eb544d59a84c78a790f0a768 | 172.30.82.85   | ACTIVE              | octavia  |
| 307b0eb5-ee04-467a-a548-fde874cd95a6 | openshift-kube-apiserver/apiserver                                    | aa0378b6eb544d59a84c78a790f0a768 | 172.30.194.43  | ACTIVE              | octavia  |
| 280204d9-5cfb-4acf-8983-1737dae5d773 | openshift-operator-lifecycle-manager/olm-operator-metrics             | aa0378b6eb544d59a84c78a790f0a768 | 172.30.248.45  | ACTIVE              | octavia  |
| 9ad9135f-0663-44fb-a984-640c0e8f639e | openshift-kube-scheduler/scheduler                                    | aa0378b6eb544d59a84c78a790f0a768 | 172.30.226.138 | ACTIVE              | octavia  |
| 646a2ead-525f-48b2-9983-9c306b0745b1 | openshift-operator-lifecycle-manager/catalog-operator-metrics         | aa0378b6eb544d59a84c78a790f0a768 | 172.30.17.72   | ACTIVE              | octavia  |
| 1bcbd619-251f-4043-8ea2-c8dd23760dae | openshift-controller-manager/controller-manager                       | aa0378b6eb544d59a84c78a790f0a768 | 172.30.62.44   | ACTIVE              | octavia  |
| e273fc88-8e1e-432c-9072-4acea769549c | openshift-kube-controller-manager/kube-controller-manager             | aa0378b6eb544d59a84c78a790f0a768 | 172.30.32.62   | ACTIVE              | octavia  |
| d035e23a-40fe-462c-af3a-98b553454e0f | openshift-operator-lifecycle-manager/v1-packages-operators-coreos-com | aa0378b6eb544d59a84c78a790f0a768 | 172.30.16.122  | ACTIVE              | octavia  |
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+----------------+---------------------+----------+





(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list
+--------------------------------------+--------------------------------------+-----------+------------+---------------+----------------+
| id                                   | loadbalancer_id                      | status    | role       | lb_network_ip | ha_ip          |
+--------------------------------------+--------------------------------------+-----------+------------+---------------+----------------+
| 09453a6a-b75e-4100-a909-b0a5bf4ac6f2 | 9f6f2c5f-4fdd-4986-8d39-1d59768206ef | ALLOCATED | STANDALONE | 172.24.0.27   | 172.30.0.1     |
| 158a48ab-867b-458f-94e3-6aba65609996 | d035e23a-40fe-462c-af3a-98b553454e0f | ALLOCATED | STANDALONE | 172.24.0.30   | 172.30.16.122  |
| 1cbfdbf2-0004-4bcc-a72a-dcde7bc61b26 | f35fa438-d40d-4454-ba00-046e74ad4b72 | ALLOCATED | STANDALONE | 172.24.0.14   | 172.30.54.49   |
| 25e45a15-fe95-4987-a390-3181d7a008ad | 9ce335fb-2565-447a-bdbb-a7be2d3f6769 | ALLOCATED | STANDALONE | 172.24.0.6    | 172.30.130.117 |
| 326c93b5-dec0-4821-bbb1-0385a81a7f07 | 6ef04bb2-4378-43aa-b793-1f3dc2a177a6 | ALLOCATED | STANDALONE | 172.24.0.17   | 172.30.62.162  |
| 3a9fac66-8751-4219-9133-dd86e02ae34f | dfc5427c-89ca-4af1-b8c6-43d9ea6b8869 | ALLOCATED | STANDALONE | 172.24.0.33   | 172.30.121.201 |
| 50635b26-d9b3-4377-b384-d170e11b92be | e273fc88-8e1e-432c-9072-4acea769549c | ALLOCATED | STANDALONE | 172.24.0.21   | 172.30.32.62   |
| 5c2f3b69-f614-4ee0-ba00-d394488bea4b | 9ad9135f-0663-44fb-a984-640c0e8f639e | ALLOCATED | STANDALONE | 172.24.0.20   | 172.30.226.138 |
| 5ff184ec-5164-4d81-9080-436c955f1797 | 314956c5-e8f7-4505-8118-eb9c664a45d0 | ALLOCATED | STANDALONE | 172.24.0.9    | 172.30.82.129  |
| 61e0968a-d4ea-4bd2-bdc8-b19c99f0f844 | 74bd7e77-97ff-4e06-89c7-f6ed68e1a988 | ALLOCATED | STANDALONE | 172.24.0.19   | 172.30.82.85   |
| 7264d58e-c2a1-4efe-b8c2-b6145bdea831 | dbbae311-14c3-4298-a228-4df5e500275e | ALLOCATED | STANDALONE | 172.24.0.16   | 172.30.0.10    |
| 7b11dc69-71a3-43c7-b680-66150134ce65 | 07489b31-730e-4f2d-b7e8-83c8b6888ad2 | ALLOCATED | STANDALONE | 172.24.0.11   | 172.30.20.213  |
| aa3c623d-be46-40fa-9731-c528ad1a744e | 307b0eb5-ee04-467a-a548-fde874cd95a6 | ALLOCATED | STANDALONE | 172.24.0.22   | 172.30.194.43  |
| c3bc608d-a70a-4b14-940c-4c2ca167b93f | 646a2ead-525f-48b2-9983-9c306b0745b1 | ALLOCATED | STANDALONE | 172.24.0.12   | 172.30.17.72   |
| d1ea9e02-2914-4af4-9e64-f97624d30113 | 46c4535d-ef50-473f-add5-1d7613639185 | ALLOCATED | STANDALONE | 172.24.0.38   | 172.30.211.35  |
| d2d2727d-e16d-4a28-b85a-3d1406008fbc | 1b1a1ffe-7048-457c-8e92-c8b53be2a30d | ALLOCATED | STANDALONE | 172.24.0.13   | 172.30.62.33   |
| f485fdbd-b0a0-4f64-b578-a1e2b4b18c33 | 28c0891a-addb-4d34-9221-83d2cd36b985 | ALLOCATED | STANDALONE | 172.24.0.15   | 172.30.118.219 |
| f52cf9f9-02ee-4a8c-9451-4c4e24f7d629 | 1bcbd619-251f-4043-8ea2-c8dd23760dae | ALLOCATED | STANDALONE | 172.24.0.42   | 172.30.62.44   |
| f90b5b9f-7dce-4a9d-859c-d414e0499088 | 854da7fc-d095-459e-a1b8-3e7de3a36bd0 | ALLOCATED | STANDALONE | 172.24.0.24   | 172.30.56.92   |
| fa6ba4d4-0df5-4934-a10c-175e75b0743c | 280204d9-5cfb-4acf-8983-1737dae5d773 | ALLOCATED | STANDALONE | 172.24.0.10   | 172.30.248.45  |
+--------------------------------------+--------------------------------------+-----------+------------+---------------+----------------+





(overcloud) [stack@undercloud-0 ~]$ oc get pods -A
NAMESPACE                                    NAME                                                     READY   STATUS      RESTARTS   AGE
openshift-apiserver-operator                 openshift-apiserver-operator-7648756bf6-snj67            1/1     Running     131        12h
openshift-cloud-credential-operator          cloud-credential-operator-cd56bbfd6-kgp9b                1/1     Running     1          12h
openshift-cluster-machine-approver           machine-approver-f99ddd99f-bst6c                         1/1     Running     0          12h
openshift-cluster-version                    cluster-version-operator-865b5b8d46-gf2j4                1/1     Running     1          12h
openshift-controller-manager-operator        openshift-controller-manager-operator-58f688fbd9-kvxg2   1/1     Running     2          12h
openshift-controller-manager                 controller-manager-fhzxf                                 1/1     Running     35         12h
openshift-controller-manager                 controller-manager-r7xtx                                 1/1     Running     40         12h
openshift-controller-manager                 controller-manager-wrf2b                                 1/1     Running     49         12h
openshift-dns-operator                       dns-operator-d864b4d65-5g68h                             1/1     Running     0          12h
openshift-dns                                dns-default-j7n9t                                        2/2     Running     0          11h
openshift-dns                                dns-default-krv6q                                        2/2     Running     0          12h
openshift-dns                                dns-default-m6k72                                        2/2     Running     0          11h
openshift-dns                                dns-default-sgxs7                                        2/2     Running     0          12h
openshift-dns                                dns-default-vl9t7                                        2/2     Running     0          12h
openshift-dns                                dns-default-w65sk                                        2/2     Running     0          11h
openshift-etcd                               etcd-member-ostest-wg9bj-master-0                        2/2     Running     0          12h
openshift-etcd                               etcd-member-ostest-wg9bj-master-1                        2/2     Running     0          12h
openshift-etcd                               etcd-member-ostest-wg9bj-master-2                        2/2     Running     0          12h
openshift-insights                           insights-operator-5568799bf7-wzpv6                       1/1     Running     0          12h
openshift-kube-apiserver-operator            kube-apiserver-operator-dc675f755-h5drq                  1/1     Running     2          12h
openshift-kube-apiserver                     installer-2-ostest-wg9bj-master-0                        0/1     Completed   0          11h
openshift-kube-apiserver                     installer-2-ostest-wg9bj-master-1                        0/1     Completed   0          12h
openshift-kube-apiserver                     installer-2-ostest-wg9bj-master-2                        0/1     Completed   0          12h
openshift-kube-apiserver                     installer-3-ostest-wg9bj-master-0                        0/1     Completed   0          11h
openshift-kube-apiserver                     installer-3-ostest-wg9bj-master-1                        0/1     Completed   0          11h
openshift-kube-apiserver                     installer-3-ostest-wg9bj-master-2                        0/1     Completed   0          11h
openshift-kube-apiserver                     kube-apiserver-ostest-wg9bj-master-0                     3/3     Running     0          11h
openshift-kube-apiserver                     kube-apiserver-ostest-wg9bj-master-1                     3/3     Running     0          11h
openshift-kube-apiserver                     kube-apiserver-ostest-wg9bj-master-2                     3/3     Running     0          11h
openshift-kube-apiserver                     revision-pruner-2-ostest-wg9bj-master-0                  0/1     Completed   0          11h
openshift-kube-apiserver                     revision-pruner-2-ostest-wg9bj-master-1                  0/1     Completed   0          11h
openshift-kube-apiserver                     revision-pruner-2-ostest-wg9bj-master-2                  0/1     Completed   0          11h
openshift-kube-apiserver                     revision-pruner-3-ostest-wg9bj-master-0                  0/1     Completed   0          11h
openshift-kube-apiserver                     revision-pruner-3-ostest-wg9bj-master-1                  0/1     Completed   0          11h
openshift-kube-apiserver                     revision-pruner-3-ostest-wg9bj-master-2                  0/1     Completed   0          11h
openshift-kube-controller-manager-operator   kube-controller-manager-operator-586f864f89-crr88        1/1     Running     2          12h
openshift-kube-controller-manager            installer-2-ostest-wg9bj-master-0                        0/1     Completed   0          12h
openshift-kube-controller-manager            installer-6-ostest-wg9bj-master-0                        0/1     Completed   0          11h
openshift-kube-controller-manager            installer-6-ostest-wg9bj-master-1                        0/1     Completed   0          12h
openshift-kube-controller-manager            installer-6-ostest-wg9bj-master-2                        0/1     Completed   0          11h
openshift-kube-controller-manager            installer-7-ostest-wg9bj-master-0                        0/1     Completed   0          11h
openshift-kube-controller-manager            installer-7-ostest-wg9bj-master-1                        0/1     Completed   0          11h
openshift-kube-controller-manager            installer-7-ostest-wg9bj-master-2                        0/1     Completed   0          11h
openshift-kube-controller-manager            kube-controller-manager-ostest-wg9bj-master-0            2/2     Running     1          11h
openshift-kube-controller-manager            kube-controller-manager-ostest-wg9bj-master-1            2/2     Running     0          11h
openshift-kube-controller-manager            kube-controller-manager-ostest-wg9bj-master-2            2/2     Running     1          11h
openshift-kube-controller-manager            revision-pruner-2-ostest-wg9bj-master-0                  0/1     Completed   0          12h
openshift-kube-controller-manager            revision-pruner-6-ostest-wg9bj-master-0                  0/1     Completed   0          11h
openshift-kube-controller-manager            revision-pruner-6-ostest-wg9bj-master-1                  0/1     Completed   0          11h
openshift-kube-controller-manager            revision-pruner-6-ostest-wg9bj-master-2                  0/1     Completed   0          11h
openshift-kube-controller-manager            revision-pruner-7-ostest-wg9bj-master-0                  0/1     Completed   0          11h
openshift-kube-controller-manager            revision-pruner-7-ostest-wg9bj-master-1                  0/1     Completed   0          11h
openshift-kube-controller-manager            revision-pruner-7-ostest-wg9bj-master-2                  0/1     Completed   0          11h
openshift-kube-scheduler-operator            openshift-kube-scheduler-operator-644d57c785-wqtxg       1/1     Running     2          12h
openshift-kube-scheduler                     installer-2-ostest-wg9bj-master-2                        0/1     Completed   0          12h
openshift-kube-scheduler                     installer-4-ostest-wg9bj-master-0                        0/1     Completed   0          12h
openshift-kube-scheduler                     installer-4-ostest-wg9bj-master-1                        0/1     Completed   0          11h
openshift-kube-scheduler                     installer-4-ostest-wg9bj-master-2                        0/1     Completed   0          11h
openshift-kube-scheduler                     openshift-kube-scheduler-ostest-wg9bj-master-0           1/1     Running     0          12h
openshift-kube-scheduler                     openshift-kube-scheduler-ostest-wg9bj-master-1           1/1     Running     1          11h
openshift-kube-scheduler                     openshift-kube-scheduler-ostest-wg9bj-master-2           1/1     Running     1          11h
openshift-kube-scheduler                     revision-pruner-2-ostest-wg9bj-master-2                  0/1     Completed   0          12h
openshift-kube-scheduler                     revision-pruner-4-ostest-wg9bj-master-0                  0/1     Completed   0          11h
openshift-kube-scheduler                     revision-pruner-4-ostest-wg9bj-master-1                  0/1     Completed   0          11h
openshift-kube-scheduler                     revision-pruner-4-ostest-wg9bj-master-2                  0/1     Completed   0          11h
openshift-kuryr                              kuryr-cni-48f6d                                          1/1     Running     0          12h
openshift-kuryr                              kuryr-cni-55hw8                                          1/1     Running     0          12h
openshift-kuryr                              kuryr-cni-6swr7                                          1/1     Running     0          12h
openshift-kuryr                              kuryr-cni-dqdc7                                          1/1     Running     0          11h
openshift-kuryr                              kuryr-cni-gszvd                                          1/1     Running     0          11h
openshift-kuryr                              kuryr-cni-z72wc                                          1/1     Running     0          11h
openshift-kuryr                              kuryr-controller-8557787669-ldqj5                        1/1     Running     3          12h
openshift-kuryr                              kuryr-dns-admission-controller-8wx5l                     1/1     Running     0          12h
openshift-kuryr                              kuryr-dns-admission-controller-lqdq6                     1/1     Running     0          12h
openshift-kuryr                              kuryr-dns-admission-controller-z2zbp                     1/1     Running     0          12h
openshift-machine-api                        machine-api-controllers-7b9c567969-rw52l                 3/3     Running     0          12h
openshift-machine-api                        machine-api-operator-668d945596-j2xvc                    1/1     Running     1          12h
openshift-machine-config-operator            etcd-quorum-guard-74fbfccd87-2mrgb                       1/1     Running     0          12h
openshift-machine-config-operator            etcd-quorum-guard-74fbfccd87-7qgkk                       1/1     Running     0          12h
openshift-machine-config-operator            etcd-quorum-guard-74fbfccd87-hmtqn                       1/1     Running     0          12h
openshift-machine-config-operator            machine-config-controller-c97d65bd7-9zt4t                1/1     Running     1          12h
openshift-machine-config-operator            machine-config-daemon-c648v                              1/1     Running     0          11h
openshift-machine-config-operator            machine-config-daemon-hrcn6                              1/1     Running     0          12h
openshift-machine-config-operator            machine-config-daemon-kkgft                              1/1     Running     0          11h
openshift-machine-config-operator            machine-config-daemon-ng22z                              1/1     Running     0          11h
openshift-machine-config-operator            machine-config-daemon-nwjs6                              1/1     Running     0          12h
openshift-machine-config-operator            machine-config-daemon-sdpsp                              1/1     Running     0          12h
openshift-machine-config-operator            machine-config-operator-8467996985-w7skh                 1/1     Running     1          12h
openshift-machine-config-operator            machine-config-server-89d5w                              1/1     Running     0          12h
openshift-machine-config-operator            machine-config-server-cbhhv                              1/1     Running     0          12h
openshift-machine-config-operator            machine-config-server-xjxfx                              1/1     Running     0          12h
openshift-multus                             multus-2qfng                                             1/1     Running     0          11h
openshift-multus                             multus-8zgpl                                             1/1     Running     0          11h
openshift-multus                             multus-admission-controller-2zbgt                        1/1     Running     0          12h
openshift-multus                             multus-admission-controller-94jtk                        1/1     Running     0          12h
openshift-multus                             multus-admission-controller-z5pjg                        1/1     Running     0          12h
openshift-multus                             multus-h7h27                                             1/1     Running     0          12h
openshift-multus                             multus-hf87d                                             1/1     Running     0          12h
openshift-multus                             multus-nc9vz                                             1/1     Running     0          12h
openshift-multus                             multus-z5sgz                                             1/1     Running     0          11h
openshift-network-operator                   network-operator-7fdc49495d-44nfj                        1/1     Running     0          12h
openshift-openstack-infra                    coredns-ostest-wg9bj-master-0                            1/1     Running     0          12h
openshift-openstack-infra                    coredns-ostest-wg9bj-master-1                            1/1     Running     0          12h
openshift-openstack-infra                    coredns-ostest-wg9bj-master-2                            1/1     Running     0          12h
openshift-openstack-infra                    coredns-ostest-wg9bj-worker-cmnlk                        1/1     Running     0          11h
openshift-openstack-infra                    coredns-ostest-wg9bj-worker-r4sj5                        1/1     Running     0          11h
openshift-openstack-infra                    coredns-ostest-wg9bj-worker-wdmxw                        1/1     Running     0          11h
openshift-openstack-infra                    haproxy-ostest-wg9bj-master-0                            2/2     Running     1          12h
openshift-openstack-infra                    haproxy-ostest-wg9bj-master-1                            2/2     Running     1          12h
openshift-openstack-infra                    haproxy-ostest-wg9bj-master-2                            2/2     Running     1          12h
openshift-openstack-infra                    keepalived-ostest-wg9bj-master-0                         1/1     Running     0          12h
openshift-openstack-infra                    keepalived-ostest-wg9bj-master-1                         1/1     Running     0          12h
openshift-openstack-infra                    keepalived-ostest-wg9bj-master-2                         1/1     Running     0          12h
openshift-openstack-infra                    keepalived-ostest-wg9bj-worker-cmnlk                     1/1     Running     0          11h
openshift-openstack-infra                    keepalived-ostest-wg9bj-worker-r4sj5                     1/1     Running     0          11h
openshift-openstack-infra                    keepalived-ostest-wg9bj-worker-wdmxw                     1/1     Running     0          11h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-master-0                     1/1     Running     0          12h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-master-1                     1/1     Running     0          12h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-master-2                     1/1     Running     0          12h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-worker-cmnlk                 1/1     Running     0          11h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-worker-r4sj5                 1/1     Running     0          11h
openshift-openstack-infra                    mdns-publisher-ostest-wg9bj-worker-wdmxw                 1/1     Running     0          11h
openshift-operator-lifecycle-manager         catalog-operator-67d65c7774-hk7pk                        1/1     Running     0          12h
openshift-operator-lifecycle-manager         olm-operator-8559ddbb5c-5dxv2                            1/1     Running     0          12h
openshift-operator-lifecycle-manager         packageserver-b5689448f-wdd4v                            1/1     Running     0          11h
openshift-operator-lifecycle-manager         packageserver-b5689448f-whhzk                            1/1     Running     0          11h
openshift-service-ca-operator                service-ca-operator-8655757d55-jz5h6                     1/1     Running     1          12h
openshift-service-ca                         apiservice-cabundle-injector-7c65c88bd9-fpm7x            1/1     Running     1          12h
openshift-service-ca                         configmap-cabundle-injector-6864bf47f7-9pfv2             1/1     Running     1          12h
openshift-service-ca                         service-serving-cert-signer-77c9c666c9-gtbsl             1/1     Running     1          12h






(shiftstack) [stack@undercloud-0 ~]$ openstack port list | grep 172.30.0.1
| 0abd15ce-2536-452a-b2b9-054421f506d5 | octavia-lb-dbbae311-14c3-4298-a228-4df5e500275e      | fa:16:3e:43:73:11 | ip_address='172.30.0.10', subnet_id='d4c54fd6-9550-4eee-b590-a46abe5a721f'    | DOWN   |
| 57d29893-fbb9-4b2a-93e0-6b07ab5dc8e7 | octavia-lb-9f6f2c5f-4fdd-4986-8d39-1d59768206ef      | fa:16:3e:2b:58:a8 | ip_address='172.30.0.1', subnet_id='d4c54fd6-9550-4eee-b590-a46abe5a721f'     | DOWN   |

--- Additional comment from Ben Bennett on 2019-12-10 14:20:50 UTC ---

Pushing to 4.4 since this is not a 4.3 regression.  When we have a fix for it, we can talk about doing a back-port.

Comment 1 Luis Tomas Bolivar 2019-12-17 16:15:17 UTC
Closing this as the bug was in OpenStack side instead of OCP side. So no need to have one for 4.4 and one for 4.3. The 4.4 bug was already moved to OSP13 (neutron): https://bugzilla.redhat.com/show_bug.cgi?id=1779654

*** This bug has been marked as a duplicate of bug 1779654 ***


Note You need to log in before you can comment on or make changes to this bug.