Bug 1737122 - Cluster Fails to install when use network type OVNKubernetes
Summary: Cluster Fails to install when use network type OVNKubernetes
Keywords:
Status: CLOSED DUPLICATE of bug 1729091
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.2.0
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: ---
: 4.2.0
Assignee: Dan Winship
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-02 17:20 UTC by Glenn West
Modified: 2019-08-13 14:36 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-13 14:36:07 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Glenn West 2019-08-02 17:20:35 UTC
Description of problem:
Cluster fails to install when using OVNKubernetes network type.


Version-Release number of selected component (if applicable):
unreleased-master-1504-g40f7fa5ed322f06d37d7dfd1e8c222499b3be718-dirty 
Tested several nightlies, that had marked as passed, but none could be installed.

How reproducible:
Very

Steps to Reproduce:
1. Modify install-config to use OVN Kubernetes
2. Run standard install
3.

Actual results:
Glenns-MacBook-Pro-2:gw.lo gwest$ oc get clusteroperators
NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
network             False       True          False      29m
Glenns-MacBook-Pro-2:gw.lo gwest$ 


apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2019-08-02T16:47:37Z"
    generation: 1
    name: version
    resourceVersion: "6106"
    selfLink: /apis/config.openshift.io/v1/clusterversions/version
    uid: 3c163626-b545-11e9-a400-0050561f2525
  spec:
    channel: stable-4.2
    clusterID: f0a9b4ea-486e-4260-b265-e1ec5d766462
    upstream: https://api.openshift.com/api/upgrades_info/v1/graph
  status:
    availableUpdates: null
    conditions:
    - lastTransitionTime: "2019-08-02T16:48:06Z"
      status: "False"
      type: Available
    - lastTransitionTime: "2019-08-02T17:15:14Z"
      message: |-
        Multiple errors are preventing progress:
        * Could not update oauthclient "console" (257 of 431): the server does not recognize this resource, check extension API servers
        * Could not update rolebinding "openshift/cluster-samples-operator-openshift-edit" (207 of 431): resource may have been deleted
        * Could not update servicemonitor "openshift-apiserver-operator/openshift-apiserver-operator" (426 of 431): the server does not recognize this resource, check extension API servers
        * Could not update servicemonitor "openshift-authentication-operator/authentication-operator" (394 of 431): the server does not recognize this resource, check extension API servers
        * Could not update servicemonitor "openshift-cluster-version/cluster-version-operator" (6 of 431): the server does not recognize this resource, check extension API servers
        * Could not update servicemonitor "openshift-controller-manager-operator/openshift-controller-manager-operator" (430 of 431): the server does not recognize this resource, check extension API servers
        * Could not update servicemonitor "openshift-image-registry/image-registry" (400 of 431): the server does not recognize this resource, check extension API servers
        * Could not update servicemonitor "openshift-kube-apiserver-operator/kube-apiserver-operator" (410 of 431): the server does not recognize this resource, check extension API servers
        * Could not update servicemonitor "openshift-kube-controller-manager-operator/kube-controller-manager-operator" (414 of 431): the server does not recognize this resource, check extension API servers
        * Could not update servicemonitor "openshift-kube-scheduler-operator/kube-scheduler-operator" (418 of 431): the server does not recognize this resource, check extension API servers
        * Could not update servicemonitor "openshift-machine-api/cluster-autoscaler-operator" (145 of 431): the server does not recognize this resource, check extension API servers
        * Could not update servicemonitor "openshift-operator-lifecycle-manager/olm-operator" (420 of 431): the server does not recognize this resource, check extension API servers
        * Could not update servicemonitor "openshift-service-catalog-apiserver-operator/openshift-service-catalog-apiserver-operator" (403 of 431): the server does not recognize this resource, check extension API servers
        * Could not update servicemonitor "openshift-service-catalog-controller-manager-operator/openshift-service-catalog-controller-manager-operator" (406 of 431): the server does not recognize this resource, check extension API servers
      reason: MultipleErrors
      status: "True"
      type: Failing
    - lastTransitionTime: "2019-08-02T16:48:06Z"
      message: 'Unable to apply 4.2.0-0.ci-2019-08-02-141506: an unknown error has
        occurred'
      reason: MultipleErrors
      status: "True"
      type: Progressing
    - lastTransitionTime: "2019-08-02T16:48:06Z"
      message: 'Unable to retrieve available updates: currently installed version
        4.2.0-0.ci-2019-08-02-141506 not found in the "stable-4.2" channel'
      reason: RemoteFailed
      status: "False"
      type: RetrievedUpdates
    desired:
      force: false
      image: registry.svc.ci.openshift.org/ocp/release@sha256:477e024e6199c0f22db049c7d01c6ce02e454cc5b6f6d06b3a1a5c6dbc138d5f
      version: 4.2.0-0.ci-2019-08-02-141506
    history:
    - completionTime: null
      image: registry.svc.ci.openshift.org/ocp/release@sha256:477e024e6199c0f22db049c7d01c6ce02e454cc5b6f6d06b3a1a5c6dbc138d5f
      startedTime: "2019-08-02T16:48:06Z"
      state: Partial
      verified: false
      version: 4.2.0-0.ci-2019-08-02-141506
    observedGeneration: 1
    versionHash: lGT54qUj3u4=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


Expected results:
Working Cluster.

Additional info:

Comment 1 Casey Callendrello 2019-08-05 12:24:20 UTC
The installer may not succeed (danw is working on some random bugs), but you should be left with a reasonably functional cluster for development work.

Comment 3 Glenn West 2019-08-05 15:43:48 UTC
Last ci build that marked green, just tested, again, it fails.


openshift-install unreleased-master-1506-g885a44213514c81507c62af9d7271e176baa5c05-dirty
built from commit 885a44213514c81507c62af9d7271e176baa5c05
release image registry.svc.ci.openshift.org/ocp/release@sha256:56d57b177645e7ffdabf776ad321fe14053c17cf80020338a7f5df5702dd24ec


Every 5.0s: oc get clusteroperators                                                                                    Glenns-MacBook-Pro-2.local: Mon Aug  5 10:43:00 2019

NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
network             False       True          False      10m




Network never will become ready.

Comment 4 Glenn West 2019-08-05 15:53:19 UTC
Glenns-MacBook-Pro-2:gw.lo gwest$ ./clusteroperatorstatus.sh 
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterOperator
  metadata:
    creationTimestamp: "2019-08-05T15:32:55Z"
    generation: 1
    name: network
    resourceVersion: "2308"
    selfLink: /apis/config.openshift.io/v1/clusteroperators/network
    uid: 4bc3a12b-b796-11e9-a820-0050561f2525
  spec: {}
  status:
    conditions:
    - lastTransitionTime: "2019-08-05T15:32:56Z"
      status: "False"
      type: Degraded
    - lastTransitionTime: "2019-08-05T15:32:56Z"
      status: "True"
      type: Upgradeable
    - lastTransitionTime: "2019-08-05T15:32:56Z"
      message: DaemonSet "openshift-multus/multus-admission-controller" is not available
        (awaiting 3 nodes)
      reason: Deploying
      status: "True"
      type: Progressing
    - lastTransitionTime: "2019-08-05T15:32:56Z"
      message: The network is starting up
      reason: Startup
      status: "False"
      type: Available

Comment 5 Glenn West 2019-08-05 16:31:23 UTC
DEBUG OpenShift Installer v4.2.0-201908010219-dirty 
DEBUG Built from commit 1f8da8a771253e74db3bde6758acac2fdbfac0d3 

Glenns-MacBook-Pro-2:gw.lo gwest$ ./clusteroperatorstatus.sh 
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterOperator
  metadata:
    creationTimestamp: "2019-08-05T16:23:58Z"
    generation: 1
    name: network
    resourceVersion: "2276"
    selfLink: /apis/config.openshift.io/v1/clusteroperators/network
    uid: 6d9839d0-b79d-11e9-9c10-0050561f2525
  spec: {}
  status:
    conditions:
    - lastTransitionTime: "2019-08-05T16:23:59Z"
      status: "False"
      type: Degraded
    - lastTransitionTime: "2019-08-05T16:23:59Z"
      status: "True"
      type: Upgradeable
    - lastTransitionTime: "2019-08-05T16:24:00Z"
      message: DaemonSet "openshift-multus/multus-admission-controller" is not available
        (awaiting 3 nodes)
      reason: Deploying
      status: "True"
      type: Progressing
    - lastTransitionTime: "2019-08-05T16:24:00Z"
      message: The network is starting up
      reason: Startup
      status: "False"
      type: Available
    extension: null
    relatedObjects:
    - group: ""

Comment 6 Glenn West 2019-08-05 17:05:45 UTC
Glenns-MacBook-Pro-2:gw.lo gwest$ oc get all --all-namespaces
NAMESPACE                                    NAME                                                         READY   STATUS              RESTARTS   AGE
openshift-apiserver-operator                 pod/openshift-apiserver-operator-5c64784d47-mkq8l            1/1     Running             1          3m19s
openshift-cloud-credential-operator          pod/cloud-credential-operator-75fcf9d7f6-lwr7r               0/1     Error               1          3m30s
openshift-cluster-machine-approver           pod/machine-approver-5fb9454f5b-j6tj4                        1/1     Running             0          3m32s
openshift-cluster-version                    pod/cluster-version-operator-79996b9db4-mv7tc                1/1     Running             0          3m13s
openshift-controller-manager-operator        pod/openshift-controller-manager-operator-76cf7864d9-4z2sv   1/1     Running             1          3m19s
openshift-dns-operator                       pod/dns-operator-67ffdc648b-58jt8                            0/1     ContainerCreating   0          6s
openshift-dns-operator                       pod/dns-operator-b5b485b4d-tjj4t                             1/1     Running             2          3m30s
openshift-etcd                               pod/etcd-member-control-plane-0                              2/2     Running             0          2m27s
openshift-etcd                               pod/etcd-member-control-plane-1                              2/2     Running             0          2m22s
openshift-etcd                               pod/etcd-member-control-plane-2                              2/2     Running             0          3m37s
openshift-insights                           pod/insights-operator-7b6c67955-4vwc7                        0/1     Terminating         1          3m25s
openshift-kube-apiserver-operator            pod/kube-apiserver-operator-65c654d984-sw6pw                 1/1     Running             1          3m19s
openshift-kube-controller-manager-operator   pod/kube-controller-manager-operator-8685fddd78-fmvl2        1/1     Running             1          3m20s
openshift-kube-scheduler-operator            pod/openshift-kube-scheduler-operator-5994687fb-7jmnf        1/1     Running             1          3m19s
openshift-machine-api                        pod/machine-api-operator-766c9b669c-qw2r5                    1/1     Running             0          2m30s
openshift-machine-config-operator            pod/machine-config-operator-57b8d4db67-rbbb9                 1/1     Running             0          3m31s
openshift-multus                             pod/multus-5c2cq                                             1/1     Running             0          3m25s
openshift-multus                             pod/multus-6w4j4                                             1/1     Running             0          3m25s
openshift-multus                             pod/multus-6xhnn                                             1/1     Running             0          3m25s
openshift-multus                             pod/multus-admission-controller-bb57p                        0/1     ContainerCreating   0          3m21s
openshift-multus                             pod/multus-admission-controller-j9cn9                        0/1     ContainerCreating   0          3m23s
openshift-multus                             pod/multus-admission-controller-nxq7m                        0/1     ContainerCreating   0          3m23s
openshift-multus                             pod/multus-bm2z9                                             1/1     Running             0          3m25s
openshift-multus                             pod/multus-j2c5b                                             1/1     Running             0          3m21s
openshift-multus                             pod/multus-jsnwv                                             1/1     Running             0          3m23s
openshift-network-operator                   pod/network-operator-6f5fb67c68-thqrb                        1/1     Running             0          3m32s
openshift-operator-lifecycle-manager         pod/catalog-operator-6dd75c7d87-btfxz                        0/1     ContainerCreating   0          3m25s
openshift-operator-lifecycle-manager         pod/olm-operator-854cffcff7-r7vfk                            0/1     ContainerCreating   0          3m25s

Comment 8 Glenn West 2019-08-05 17:10:04 UTC
Glenns-MacBook-Pro-2:gw.lo gwest$ oc get all --all-namespaces
NAMESPACE                                    NAME                                                         READY   STATUS              RESTARTS   AGE
openshift-apiserver-operator                 pod/openshift-apiserver-operator-5c64784d47-mkq8l            1/1     Running             1          3m19s
openshift-cloud-credential-operator          pod/cloud-credential-operator-75fcf9d7f6-lwr7r               0/1     Error               1          3m30s
openshift-cluster-machine-approver           pod/machine-approver-5fb9454f5b-j6tj4                        1/1     Running             0          3m32s
openshift-cluster-version                    pod/cluster-version-operator-79996b9db4-mv7tc                1/1     Running             0          3m13s
openshift-controller-manager-operator        pod/openshift-controller-manager-operator-76cf7864d9-4z2sv   1/1     Running             1          3m19s
openshift-dns-operator                       pod/dns-operator-67ffdc648b-58jt8                            0/1     ContainerCreating   0          6s
openshift-dns-operator                       pod/dns-operator-b5b485b4d-tjj4t                             1/1     Running             2          3m30s
openshift-etcd                               pod/etcd-member-control-plane-0                              2/2     Running             0          2m27s
openshift-etcd                               pod/etcd-member-control-plane-1                              2/2     Running             0          2m22s
openshift-etcd                               pod/etcd-member-control-plane-2                              2/2     Running             0          3m37s
openshift-insights                           pod/insights-operator-7b6c67955-4vwc7                        0/1     Terminating         1          3m25s
openshift-kube-apiserver-operator            pod/kube-apiserver-operator-65c654d984-sw6pw                 1/1     Running             1          3m19s
openshift-kube-controller-manager-operator   pod/kube-controller-manager-operator-8685fddd78-fmvl2        1/1     Running             1          3m20s
openshift-kube-scheduler-operator            pod/openshift-kube-scheduler-operator-5994687fb-7jmnf        1/1     Running             1          3m19s
openshift-machine-api                        pod/machine-api-operator-766c9b669c-qw2r5                    1/1     Running             0          2m30s
openshift-machine-config-operator            pod/machine-config-operator-57b8d4db67-rbbb9                 1/1     Running             0          3m31s
openshift-multus                             pod/multus-5c2cq                                             1/1     Running             0          3m25s
openshift-multus                             pod/multus-6w4j4                                             1/1     Running             0          3m25s
openshift-multus                             pod/multus-6xhnn                                             1/1     Running             0          3m25s
openshift-multus                             pod/multus-admission-controller-bb57p                        0/1     ContainerCreating   0          3m21s
openshift-multus                             pod/multus-admission-controller-j9cn9                        0/1     ContainerCreating   0          3m23s
openshift-multus                             pod/multus-admission-controller-nxq7m                        0/1     ContainerCreating   0          3m23s
openshift-multus                             pod/multus-bm2z9                                             1/1     Running             0          3m25s
openshift-multus                             pod/multus-j2c5b                                             1/1     Running             0          3m21s
openshift-multus                             pod/multus-jsnwv                                             1/1     Running             0          3m23s
openshift-network-operator                   pod/network-operator-6f5fb67c68-thqrb                        1/1     Running             0          3m32s
openshift-operator-lifecycle-manager         pod/catalog-operator-6dd75c7d87-btfxz                        0/1     ContainerCreating   0          3m25s
openshift-operator-lifecycle-manager         pod/olm-operator-854cffcff7-r7vfk                            0/1     ContainerCreating   0          3m25s
openshift-ovn-kubernetes                     pod/ovnkube-master-7fb7c59fd6-7v6t8                          4/4     Running             0          3m13s
openshift-ovn-kubernetes                     pod/ovnkube-node-cdt97                                       3/3     Running             0          3m12s
openshift-ovn-kubernetes                     pod/ovnkube-node-fbm2d                                       3/3     Running             0          3m11s
openshift-ovn-kubernetes                     pod/ovnkube-node-rx55q                                       3/3     Running             0          3m11s
openshift-ovn-kubernetes                     pod/ovnkube-node-sf2mh                                       3/3     Running             0          3m12s
openshift-ovn-kubernetes                     pod/ovnkube-node-swmn2                                       3/3     Running             1          3m11s
openshift-ovn-kubernetes                     pod/ovnkube-node-tj92z                                       3/3     Running             0          3m12s
openshift-service-ca-operator                pod/service-ca-operator-667d77b8b9-c24fp                     1/1     Running             1          3m21s
openshift-vertical-pod-autoscaler            pod/vertical-pod-autoscaler-operator-5fc74c8579-nv6gv        1/1     Running             2          3m26s

NAMESPACE                                               NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
default                                                 service/kubernetes                    ClusterIP   172.30.0.1       <none>        443/TCP             4m16s
openshift-apiserver-operator                            service/metrics                       ClusterIP   172.30.103.121   <none>        443/TCP             3m21s
openshift-authentication-operator                       service/metrics                       ClusterIP   172.30.201.244   <none>        443/TCP             3m23s
openshift-cloud-credential-operator                     service/controller-manager-service    ClusterIP   172.30.174.49    <none>        443/TCP             3m31s
openshift-cluster-version                               service/cluster-version-operator      ClusterIP   172.30.142.233   <none>        9099/TCP            4m20s
openshift-controller-manager-operator                   service/metrics                       ClusterIP   172.30.24.116    <none>        443/TCP             3m22s
openshift-etcd                                          service/etcd                          ClusterIP   172.30.140.110   <none>        2379/TCP,9979/TCP   4m17s
openshift-etcd                                          service/host-etcd                     ClusterIP   None             <none>        2379/TCP            4m8s
openshift-kube-apiserver-operator                       service/metrics                       ClusterIP   172.30.88.35     <none>        443/TCP             3m23s
openshift-kube-controller-manager-operator              service/metrics                       ClusterIP   172.30.151.164   <none>        443/TCP             3m23s
openshift-kube-scheduler-operator                       service/metrics                       ClusterIP   172.30.19.0      <none>        443/TCP             3m23s
openshift-machine-api                                   service/cluster-autoscaler-operator   ClusterIP   172.30.37.67     <none>        443/TCP,8080/TCP    3m27s
openshift-machine-api                                   service/machine-api-operator          ClusterIP   172.30.124.45    <none>        8080/TCP            2m34s
openshift-multus                                        service/multus-admission-controller   ClusterIP   172.30.254.133   <none>        443/TCP             3m25s
openshift-operator-lifecycle-manager                    service/catalog-operator-metrics      ClusterIP   172.30.61.49     <none>        8081/TCP            3m30s
openshift-operator-lifecycle-manager                    service/olm-operator-metrics          ClusterIP   172.30.176.100   <none>        8081/TCP            3m31s
openshift-ovn-kubernetes                                service/ovnkube-db                    ClusterIP   None             <none>        6641/TCP,6642/TCP   3m14s
openshift-service-catalog-apiserver-operator            service/metrics                       ClusterIP   172.30.147.187   <none>        443/TCP             3m20s
openshift-service-catalog-controller-manager-operator   service/metrics                       ClusterIP   172.30.5.48      <none>        443/TCP             3m22s

NAMESPACE                  NAME                                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
openshift-multus           daemonset.apps/multus                        6         6         6       6            6           kubernetes.io/os=linux            3m25s
openshift-multus           daemonset.apps/multus-admission-controller   3         3         0       3            0           node-role.kubernetes.io/master=   3m23s
openshift-ovn-kubernetes   daemonset.apps/ovnkube-node                  6         6         6       6            6           beta.kubernetes.io/os=linux       3m13s

NAMESPACE                                               NAME                                                                    READY   UP-TO-DATE   AVAILABLE   AGE
openshift-apiserver-operator                            deployment.apps/openshift-apiserver-operator                            1/1     1            1           3m20s
openshift-authentication-operator                       deployment.apps/authentication-operator                                 0/1     0            0           2m52s
openshift-cloud-credential-operator                     deployment.apps/cloud-credential-operator                               0/1     1            0           3m30s
openshift-cluster-machine-approver                      deployment.apps/machine-approver                                        1/1     1            1           3m32s
openshift-cluster-node-tuning-operator                  deployment.apps/cluster-node-tuning-operator                            0/1     0            0           3m31s
openshift-cluster-storage-operator                      deployment.apps/cluster-storage-operator                                0/1     0            0           3m31s
openshift-cluster-version                               deployment.apps/cluster-version-operator                                1/1     1            1           4m13s
openshift-controller-manager-operator                   deployment.apps/openshift-controller-manager-operator                   1/1     1            1           3m19s
openshift-dns-operator                                  deployment.apps/dns-operator                                            1/1     1            1           3m30s
openshift-image-registry                                deployment.apps/cluster-image-registry-operator                         0/1     0            0           2m49s
openshift-ingress-operator                              deployment.apps/ingress-operator                                        0/1     0            0           2m49s
openshift-insights                                      deployment.apps/insights-operator                                       0/1     0            0           3m28s
openshift-kube-apiserver-operator                       deployment.apps/kube-apiserver-operator                                 1/1     1            1           3m20s
openshift-kube-controller-manager-operator              deployment.apps/kube-controller-manager-operator                        1/1     1            1           3m21s
openshift-kube-scheduler-operator                       deployment.apps/openshift-kube-scheduler-operator                       1/1     1            1           3m20s
openshift-machine-api                                   deployment.apps/machine-api-operator                                    1/1     1            1           2m30s
openshift-machine-config-operator                       deployment.apps/machine-config-operator                                 1/1     1            1           3m31s
openshift-marketplace                                   deployment.apps/marketplace-operator                                    0/1     0            0           3m29s
openshift-monitoring                                    deployment.apps/cluster-monitoring-operator                             0/1     0            0           3m23s
openshift-network-operator                              deployment.apps/network-operator                                        1/1     1            1           3m32s
openshift-operator-lifecycle-manager                    deployment.apps/catalog-operator                                        0/1     1            0           3m25s
openshift-operator-lifecycle-manager                    deployment.apps/olm-operator                                            0/1     1            0           3m25s
openshift-ovn-kubernetes                                deployment.apps/ovnkube-master                                          1/1     1            1           3m14s
openshift-service-ca-operator                           deployment.apps/service-ca-operator                                     1/1     1            1           3m22s
openshift-service-catalog-apiserver-operator            deployment.apps/openshift-service-catalog-apiserver-operator            0/1     0            0           3m19s
openshift-service-catalog-controller-manager-operator   deployment.apps/openshift-service-catalog-controller-manager-operator   0/1     0            0           3m20s
openshift-vertical-pod-autoscaler                       deployment.apps/vertical-pod-autoscaler-operator                        1/1     1            1           3m26s

NAMESPACE                                               NAME                                                                               DESIRED   CURRENT   READY   AGE
openshift-apiserver-operator                            replicaset.apps/openshift-apiserver-operator-5c64784d47                            1         1         1       3m20s
openshift-authentication-operator                       replicaset.apps/authentication-operator-6fc4b64446                                 1         0         0       2m52s
openshift-cloud-credential-operator                     replicaset.apps/cloud-credential-operator-75fcf9d7f6                               1         1         0       3m30s
openshift-cluster-machine-approver                      replicaset.apps/machine-approver-5fb9454f5b                                        1         1         1       3m32s
openshift-cluster-node-tuning-operator                  replicaset.apps/cluster-node-tuning-operator-6d64dbb9fc                            1         0         0       3m31s
openshift-cluster-storage-operator                      replicaset.apps/cluster-storage-operator-75d8554649                                1         0         0       3m31s
openshift-cluster-version                               replicaset.apps/cluster-version-operator-79996b9db4                                1         1         1       3m13s
openshift-cluster-version                               replicaset.apps/cluster-version-operator-7bd9f465c4                                0         0         0       4m13s
openshift-controller-manager-operator                   replicaset.apps/openshift-controller-manager-operator-76cf7864d9                   1         1         1       3m19s
openshift-dns-operator                                  replicaset.apps/dns-operator-67ffdc648b                                            1         1         0       6s
openshift-dns-operator                                  replicaset.apps/dns-operator-b5b485b4d                                             1         1         1       3m30s
openshift-image-registry                                replicaset.apps/cluster-image-registry-operator-784574746d                         1         0         0       2m49s
openshift-ingress-operator                              replicaset.apps/ingress-operator-7f79b89c57                                        1         0         0       2m49s
openshift-insights                                      replicaset.apps/insights-operator-64c548fbfb                                       0         0         0       3m27s
openshift-insights                                      replicaset.apps/insights-operator-7b6c67955                                        0         0         0       3m25s
openshift-kube-apiserver-operator                       replicaset.apps/kube-apiserver-operator-65c654d984                                 1         1         1       3m20s
openshift-kube-controller-manager-operator              replicaset.apps/kube-controller-manager-operator-8685fddd78                        1         1         1       3m20s
openshift-kube-scheduler-operator                       replicaset.apps/openshift-kube-scheduler-operator-5994687fb                        1         1         1       3m20s
openshift-machine-api                                   replicaset.apps/machine-api-operator-766c9b669c                                    1         1         1       2m30s
openshift-machine-config-operator                       replicaset.apps/machine-config-operator-57b8d4db67                                 1         1         1       3m31s
openshift-marketplace                                   replicaset.apps/marketplace-operator-98bb96567                                     1         0         0       3m29s
openshift-monitoring                                    replicaset.apps/cluster-monitoring-operator-dd65594f9                              1         0         0       3m23s
openshift-network-operator                              replicaset.apps/network-operator-6f5fb67c68                                        1         1         1       3m32s
openshift-operator-lifecycle-manager                    replicaset.apps/catalog-operator-6dd75c7d87                                        1         1         0       3m25s
openshift-operator-lifecycle-manager                    replicaset.apps/olm-operator-854cffcff7                                            1         1         0       3m25s
openshift-ovn-kubernetes                                replicaset.apps/ovnkube-master-7fb7c59fd6                                          1         1         1       3m13s
openshift-service-ca-operator                           replicaset.apps/service-ca-operator-667d77b8b9                                     1         1         1       3m22s
openshift-service-catalog-apiserver-operator            replicaset.apps/openshift-service-catalog-apiserver-operator-96f649554             1         0         0       3m19s
openshift-service-catalog-controller-manager-operator   replicaset.apps/openshift-service-catalog-controller-manager-operator-79c97695c4   1         0         0       3m20s
openshift-vertical-pod-autoscaler                       replicaset.apps/vertical-pod-autoscaler-operator-5fc74c8579                        1         1         1       3m26s
Glenns-MacBook-Pro-2:gw.lo gwest$ oc get all --all-namespaces
NAMESPACE                                    NAME                                                         READY   STATUS              RESTARTS   AGE
openshift-apiserver-operator                 pod/openshift-apiserver-operator-5c64784d47-mkq8l            0/1     CrashLoopBackOff    3          7m35s
openshift-cloud-credential-operator          pod/cloud-credential-operator-75fcf9d7f6-lwr7r               1/1     Running             5          7m46s
openshift-cluster-machine-approver           pod/machine-approver-5fb9454f5b-j6tj4                        1/1     Running             0          7m48s
openshift-cluster-version                    pod/cluster-version-operator-79996b9db4-mv7tc                1/1     Running             0          7m29s
openshift-controller-manager-operator        pod/openshift-controller-manager-operator-76cf7864d9-4z2sv   0/1     CrashLoopBackOff    3          7m35s
openshift-dns-operator                       pod/dns-operator-67ffdc648b-58jt8                            0/1     Error               4          4m22s
openshift-etcd                               pod/etcd-member-control-plane-0                              2/2     Running             0          6m43s
openshift-etcd                               pod/etcd-member-control-plane-1                              2/2     Running             0          6m38s
openshift-etcd                               pod/etcd-member-control-plane-2                              2/2     Running             0          7m53s
openshift-insights                           pod/insights-operator-64c548fbfb-ppd9j                       1/1     Running             0          26s
openshift-kube-apiserver-operator            pod/kube-apiserver-operator-65c654d984-sw6pw                 0/1     CrashLoopBackOff    3          7m35s
openshift-kube-controller-manager-operator   pod/kube-controller-manager-operator-8685fddd78-fmvl2        0/1     CrashLoopBackOff    3          7m36s
openshift-kube-scheduler-operator            pod/openshift-kube-scheduler-operator-5994687fb-7jmnf        0/1     CrashLoopBackOff    3          7m35s
openshift-machine-api                        pod/machine-api-operator-766c9b669c-qw2r5                    1/1     Running             0          6m46s
openshift-machine-config-operator            pod/machine-config-operator-57b8d4db67-rbbb9                 1/1     Running             0          7m47s
openshift-multus                             pod/multus-5c2cq                                             1/1     Running             0          7m41s
openshift-multus                             pod/multus-6w4j4                                             1/1     Running             0          7m41s
openshift-multus                             pod/multus-6xhnn                                             1/1     Running             0          7m41s
openshift-multus                             pod/multus-admission-controller-bb57p                        0/1     ContainerCreating   0          7m37s
openshift-multus                             pod/multus-admission-controller-j9cn9                        0/1     ContainerCreating   0          7m39s
openshift-multus                             pod/multus-admission-controller-nxq7m                        0/1     ContainerCreating   0          7m39s
openshift-multus                             pod/multus-bm2z9                                             1/1     Running             0          7m41s
openshift-multus                             pod/multus-j2c5b                                             1/1     Running             0          7m37s
openshift-multus                             pod/multus-jsnwv                                             1/1     Running             0          7m39s
openshift-network-operator                   pod/network-operator-6f5fb67c68-thqrb                        1/1     Running             0          7m48s
openshift-operator-lifecycle-manager         pod/catalog-operator-6dd75c7d87-btfxz                        0/1     ContainerCreating   0          7m41s
openshift-operator-lifecycle-manager         pod/olm-operator-854cffcff7-r7vfk                            0/1     ContainerCreating   0          7m41s
openshift-ovn-kubernetes                     pod/ovnkube-master-7fb7c59fd6-7v6t8                          4/4     Running             0          7m29s
openshift-ovn-kubernetes                     pod/ovnkube-node-cdt97                                       3/3     Running             0          7m28s
openshift-ovn-kubernetes                     pod/ovnkube-node-fbm2d                                       3/3     Running             0          7m27s
openshift-ovn-kubernetes                     pod/ovnkube-node-rx55q                                       3/3     Running             0          7m27s
openshift-ovn-kubernetes                     pod/ovnkube-node-sf2mh                                       3/3     Running             0          7m28s
openshift-ovn-kubernetes                     pod/ovnkube-node-swmn2                                       3/3     Running             1          7m27s
openshift-ovn-kubernetes                     pod/ovnkube-node-tj92z                                       3/3     Running             0          7m28s
openshift-service-ca-operator                pod/service-ca-operator-667d77b8b9-c24fp                     1/1     Running             4          7m37s
openshift-vertical-pod-autoscaler            pod/vertical-pod-autoscaler-operator-5fc74c8579-nv6gv        1/1     Running             5          7m42s

NAMESPACE                                               NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
default                                                 service/kubernetes                    ClusterIP   172.30.0.1       <none>        443/TCP             8m32s
openshift-apiserver-operator                            service/metrics                       ClusterIP   172.30.103.121   <none>        443/TCP             7m37s
openshift-authentication-operator                       service/metrics                       ClusterIP   172.30.201.244   <none>        443/TCP             7m39s
openshift-cloud-credential-operator                     service/controller-manager-service    ClusterIP   172.30.174.49    <none>        443/TCP             7m47s
openshift-cluster-version                               service/cluster-version-operator      ClusterIP   172.30.142.233   <none>        9099/TCP            8m36s
openshift-controller-manager-operator                   service/metrics                       ClusterIP   172.30.24.116    <none>        443/TCP             7m38s
openshift-etcd                                          service/etcd                          ClusterIP   172.30.140.110   <none>        2379/TCP,9979/TCP   8m33s
openshift-etcd                                          service/host-etcd                     ClusterIP   None             <none>        2379/TCP            8m24s
openshift-kube-apiserver-operator                       service/metrics                       ClusterIP   172.30.88.35     <none>        443/TCP             7m39s
openshift-kube-controller-manager-operator              service/metrics                       ClusterIP   172.30.151.164   <none>        443/TCP             7m39s
openshift-kube-scheduler-operator                       service/metrics                       ClusterIP   172.30.19.0      <none>        443/TCP             7m39s
openshift-machine-api                                   service/cluster-autoscaler-operator   ClusterIP   172.30.37.67     <none>        443/TCP,8080/TCP    7m43s
openshift-machine-api                                   service/machine-api-operator          ClusterIP   172.30.124.45    <none>        8080/TCP            6m50s
openshift-multus                                        service/multus-admission-controller   ClusterIP   172.30.254.133   <none>        443/TCP             7m41s
openshift-operator-lifecycle-manager                    service/catalog-operator-metrics      ClusterIP   172.30.61.49     <none>        8081/TCP            7m46s
openshift-operator-lifecycle-manager                    service/olm-operator-metrics          ClusterIP   172.30.176.100   <none>        8081/TCP            7m47s
openshift-ovn-kubernetes                                service/ovnkube-db                    ClusterIP   None             <none>        6641/TCP,6642/TCP   7m30s
openshift-service-catalog-apiserver-operator            service/metrics                       ClusterIP   172.30.147.187   <none>        443/TCP             7m36s
openshift-service-catalog-controller-manager-operator   service/metrics                       ClusterIP   172.30.5.48      <none>        443/TCP             7m38s

NAMESPACE                  NAME                                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
openshift-multus           daemonset.apps/multus                        6         6         6       6            6           kubernetes.io/os=linux            7m41s
openshift-multus           daemonset.apps/multus-admission-controller   3         3         0       3            0           node-role.kubernetes.io/master=   7m39s
openshift-ovn-kubernetes   daemonset.apps/ovnkube-node                  6         6         6       6            6           beta.kubernetes.io/os=linux       7m29s

NAMESPACE                                               NAME                                                                    READY   UP-TO-DATE   AVAILABLE   AGE
openshift-apiserver-operator                            deployment.apps/openshift-apiserver-operator                            0/1     1            0           7m36s
openshift-authentication-operator                       deployment.apps/authentication-operator                                 0/1     0            0           7m8s
openshift-cloud-credential-operator                     deployment.apps/cloud-credential-operator                               1/1     1            1           7m46s
openshift-cluster-machine-approver                      deployment.apps/machine-approver                                        1/1     1            1           7m48s
openshift-cluster-node-tuning-operator                  deployment.apps/cluster-node-tuning-operator                            0/1     0            0           7m47s
openshift-cluster-storage-operator                      deployment.apps/cluster-storage-operator                                0/1     0            0           7m47s
openshift-cluster-version                               deployment.apps/cluster-version-operator                                1/1     1            1           8m29s
openshift-controller-manager-operator                   deployment.apps/openshift-controller-manager-operator                   0/1     1            0           7m35s
openshift-dns-operator                                  deployment.apps/dns-operator                                            0/1     1            0           7m46s
openshift-image-registry                                deployment.apps/cluster-image-registry-operator                         0/1     0            0           7m5s
openshift-ingress-operator                              deployment.apps/ingress-operator                                        0/1     0            0           7m5s
openshift-insights                                      deployment.apps/insights-operator                                       1/1     1            1           7m44s
openshift-kube-apiserver-operator                       deployment.apps/kube-apiserver-operator                                 0/1     1            0           7m36s
openshift-kube-controller-manager-operator              deployment.apps/kube-controller-manager-operator                        0/1     1            0           7m37s
openshift-kube-scheduler-operator                       deployment.apps/openshift-kube-scheduler-operator                       0/1     1            0           7m36s
openshift-machine-api                                   deployment.apps/machine-api-operator                                    1/1     1            1           6m46s
openshift-machine-config-operator                       deployment.apps/machine-config-operator                                 1/1     1            1           7m47s
openshift-marketplace                                   deployment.apps/marketplace-operator                                    0/1     0            0           7m45s
openshift-monitoring                                    deployment.apps/cluster-monitoring-operator                             0/1     0            0           7m39s
openshift-network-operator                              deployment.apps/network-operator                                        1/1     1            1           7m48s
openshift-operator-lifecycle-manager                    deployment.apps/catalog-operator                                        0/1     1            0           7m41s
openshift-operator-lifecycle-manager                    deployment.apps/olm-operator                                            0/1     1            0           7m41s
openshift-ovn-kubernetes                                deployment.apps/ovnkube-master                                          1/1     1            1           7m30s
openshift-service-ca-operator                           deployment.apps/service-ca-operator                                     1/1     1            1           7m38s
openshift-service-catalog-apiserver-operator            deployment.apps/openshift-service-catalog-apiserver-operator            0/1     0            0           7m35s
openshift-service-catalog-controller-manager-operator   deployment.apps/openshift-service-catalog-controller-manager-operator   0/1     0            0           7m36s
openshift-vertical-pod-autoscaler                       deployment.apps/vertical-pod-autoscaler-operator                        1/1     1            1           7m42s

NAMESPACE                                               NAME                                                                               DESIRED   CURRENT   READY   AGE
openshift-apiserver-operator                            replicaset.apps/openshift-apiserver-operator-5c64784d47                            1         1         0       7m36s
openshift-authentication-operator                       replicaset.apps/authentication-operator-6fc4b64446                                 1         0         0       7m8s
openshift-cloud-credential-operator                     replicaset.apps/cloud-credential-operator-75fcf9d7f6                               1         1         1       7m46s
openshift-cluster-machine-approver                      replicaset.apps/machine-approver-5fb9454f5b                                        1         1         1       7m48s
openshift-cluster-node-tuning-operator                  replicaset.apps/cluster-node-tuning-operator-6d64dbb9fc                            1         0         0       7m47s
openshift-cluster-storage-operator                      replicaset.apps/cluster-storage-operator-75d8554649                                1         0         0       7m47s
openshift-cluster-version                               replicaset.apps/cluster-version-operator-79996b9db4                                1         1         1       7m29s
openshift-cluster-version                               replicaset.apps/cluster-version-operator-7bd9f465c4                                0         0         0       8m29s
openshift-controller-manager-operator                   replicaset.apps/openshift-controller-manager-operator-76cf7864d9                   1         1         0       7m35s
openshift-dns-operator                                  replicaset.apps/dns-operator-67ffdc648b                                            1         1         0       4m22s
openshift-dns-operator                                  replicaset.apps/dns-operator-b5b485b4d                                             0         0         0       7m46s
openshift-image-registry                                replicaset.apps/cluster-image-registry-operator-784574746d                         1         0         0       7m5s
openshift-ingress-operator                              replicaset.apps/ingress-operator-7f79b89c57                                        1         0         0       7m5s
openshift-insights                                      replicaset.apps/insights-operator-64c548fbfb                                       1         1         1       7m43s
openshift-insights                                      replicaset.apps/insights-operator-7b6c67955                                        0         0         0       7m41s
openshift-kube-apiserver-operator                       replicaset.apps/kube-apiserver-operator-65c654d984                                 1         1         0       7m36s
openshift-kube-controller-manager-operator              replicaset.apps/kube-controller-manager-operator-8685fddd78                        1         1         0       7m36s
openshift-kube-scheduler-operator                       replicaset.apps/openshift-kube-scheduler-operator-5994687fb                        1         1         0       7m36s
openshift-machine-api                                   replicaset.apps/machine-api-operator-766c9b669c                                    1         1         1       6m46s
openshift-machine-config-operator                       replicaset.apps/machine-config-operator-57b8d4db67                                 1         1         1       7m47s
openshift-marketplace                                   replicaset.apps/marketplace-operator-98bb96567                                     1         0         0       7m45s
openshift-monitoring                                    replicaset.apps/cluster-monitoring-operator-dd65594f9                              1         0         0       7m39s
openshift-network-operator                              replicaset.apps/network-operator-6f5fb67c68                                        1         1         1       7m48s
openshift-operator-lifecycle-manager                    replicaset.apps/catalog-operator-6dd75c7d87                                        1         1         0       7m41s
openshift-operator-lifecycle-manager                    replicaset.apps/olm-operator-854cffcff7                                            1         1         0       7m41s
openshift-ovn-kubernetes                                replicaset.apps/ovnkube-master-7fb7c59fd6                                          1         1         1       7m29s
openshift-service-ca-operator                           replicaset.apps/service-ca-operator-667d77b8b9                                     1         1         1       7m38s
openshift-service-catalog-apiserver-operator            replicaset.apps/openshift-service-catalog-apiserver-operator-96f649554             1         0         0       7m35s
openshift-service-catalog-controller-manager-operator   replicaset.apps/openshift-service-catalog-controller-manager-operator-79c97695c4   1         0         0       7m36s
openshift-vertical-pod-autoscaler                       replicaset.apps/vertical-pod-autoscaler-operator-5fc74c8579                        1         1         1       7m42s
Glenns-MacBook-Pro-2:gw.lo gwest$

Comment 10 Glenn West 2019-08-05 20:39:51 UTC
DEBUG OpenShift Installer unreleased-master-1508-g5e9cde9c162b313eed46ad964c9211f90e633681-dirty Glenns-MacBook-Pro-2:gw.lo gwest$ ./clusteroperatorstatus.sh 
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterOperator
  metadata:
    creationTimestamp: "2019-08-05T20:29:33Z"
    generation: 1
    name: network
    resourceVersion: "2258"
    selfLink: /apis/config.openshift.io/v1/clusteroperators/network
    uid: bc437148-b7bf-11e9-b235-0050561f2525
  spec: {}
  status:
    conditions:
    - lastTransitionTime: "2019-08-05T20:29:34Z"
      status: "False"
      type: Degraded
    - lastTransitionTime: "2019-08-05T20:29:34Z"
      status: "True"
      type: Upgradeable
    - lastTransitionTime: "2019-08-05T20:29:34Z"
      message: DaemonSet "openshift-multus/multus-admission-controller" is not available
        (awaiting 3 nodes)
      reason: Deploying
      status: "True"
      type: Progressing
    - lastTransitionTime: "2019-08-05T20:29:34Z"
      message: The network is starting up
      reason: Startup
      status: "False"
      type: Available
    extension: null

Comment 11 Glenn West 2019-08-05 20:54:59 UTC
openshift-install unreleased-master-1508-g5e9cde9c162b313eed46ad964c9211f90e633681-dirty
built from commit 5e9cde9c162b313eed46ad964c9211f90e633681
release image registry.svc.ci.openshift.org/ocp/release@sha256:c8384a27c7c785cfeed564f9fc0373a132b25fb83a5d72d29afdf477518a42ab
Glenns-MacBook-Pro-2:gw.lo gwest$ 

Glenns-MacBook-Pro-2:gw.lo gwest$ oc get all
NAME                                    READY   STATUS              RESTARTS   AGE
pod/multus-267bc                        1/1     Running             0          23m
pod/multus-admission-controller-7tkgl   0/1     ContainerCreating   0          23m
pod/multus-admission-controller-h6846   0/1     ContainerCreating   0          23m
pod/multus-admission-controller-ztfdm   0/1     ContainerCreating   0          23m
pod/multus-dtrzk                        1/1     Running             0          23m
pod/multus-f8gwk                        1/1     Running             0          23m
pod/multus-fqnq6                        1/1     Running             0          23m
pod/multus-r7tzd                        1/1     Running             0          23m
pod/multus-sf7tq                        1/1     Running             0          23m

NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/multus-admission-controller   ClusterIP   172.30.97.143   <none>        443/TCP   23m

NAME                                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
daemonset.apps/multus                        6         6         6       6            6           kubernetes.io/os=linux            23m
daemonset.apps/multus-admission-controller   3         3         0       3            0           node-role.kubernetes.io/master=   23m


Exploring cultus not starting:

Glenns-MacBook-Pro-2:gw.lo gwest$ oc logs pod/multus-admission-controller-7tkgl
Error from server (BadRequest): container "multus-admission-controller" in pod "multus-admission-controller-7tkgl" is waiting to start: ContainerCreating
Glenns-MacBook-Pro-2:gw.lo gwest$ oc describe pod/multus-admission-controller-7tkgl
Name:               multus-admission-controller-7tkgl
Namespace:          openshift-multus
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               control-plane-2/192.168.1.203
Start Time:         Mon, 05 Aug 2019 15:29:37 -0500
Labels:             app=multus-admission-controller
                    component=network
                    controller-revision-hash=5fd5775d87
                    namespace=openshift-multus
                    openshift.io/component=network
                    pod-template-generation=1
                    type=infra
Annotations:        ovn: {"ip_address":"10.131.0.3/23", "mac_address":"ee:a5:d1:83:00:04", "gateway_ip": "10.131.0.1"}
Status:             Pending
IP:                 
Controlled By:      DaemonSet/multus-admission-controller
Containers:
  multus-admission-controller:
    Container ID:  
    Image:         registry.svc.ci.openshift.org/ocp/4.2-2019-08-05-175235@sha256:d141522c3c19f598a50765dcdbcd56f70c4c3ccae0004d5922908eebba7b6205
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/webhook
    Args:
      -bind-address=0.0.0.0
      -port=6443
      -tls-private-key-file=/etc/webhook/tls.key
      -tls-cert-file=/etc/webhook/tls.crt
      -alsologtostderr=true
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        10m
    Environment:  <none>
    Mounts:
      /etc/webhook from webhook-certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from multus-token-vslwd (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  webhook-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  multus-admission-controller-secret
    Optional:    false
  multus-token-vslwd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  multus-token-vslwd
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason           Age                  From                      Message
  ----     ------           ----                 ----                      -------
  Normal   Scheduled        21m                  default-scheduler         Successfully assigned openshift-multus/multus-admission-controller-7tkgl to control-plane-2
  Warning  NetworkNotReady  21m (x18 over 21m)   kubelet, control-plane-2  network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network
  Warning  FailedMount      6m42s (x6 over 18m)  kubelet, control-plane-2  Unable to mount volumes for pod "multus-admission-controller-7tkgl_openshift-multus(be71faf6-b7bf-11e9-b235-0050561f2525)": timeout expired waiting for volumes to attach or mount for pod "openshift-multus"/"multus-admission-controller-7tkgl". list of unmounted volumes=[webhook-certs]. list of unattached volumes=[webhook-certs multus-token-vslwd]
  Warning  FailedMount      83s (x18 over 21m)   kubelet, control-plane-2  MountVolume.SetUp failed for volume "webhook-certs" : secrets "multus-admission-controller-secret" not found
Glenns-MacBook-Pro-2:gw.lo gwest$

Comment 12 Glenn West 2019-08-05 21:36:47 UTC
Glenns-MacBook-Pro-2:gw.lo gwest$ openshift-install version
openshift-install unreleased-master-1508-g5e9cde9c162b313eed46ad964c9211f90e633681-dirty
built from commit 5e9cde9c162b313eed46ad964c9211f90e633681
release image registry.svc.ci.openshift.org/ocp/release@sha256:6ca9b56129b2a67686c6157693038ea9b2c227e77a9def1b962aad9e79080b76
Glenns-MacBook-Pro-2:gw.lo gwest$ 

Glenns-MacBook-Pro-2:gw.lo gwest$ oc project openshift-multus 
Now using project "openshift-multus" on server "https://api.gw.lo:6443".
Glenns-MacBook-Pro-2:gw.lo gwest$ oc get all
NAME                                    READY   STATUS              RESTARTS   AGE
pod/multus-22hfr                        1/1     Running             0          5m44s
pod/multus-2fjtj                        1/1     Running             0          5m44s
pod/multus-4pfhs                        1/1     Running             0          5m44s
pod/multus-6zmnt                        1/1     Running             0          5m44s
pod/multus-admission-controller-8kjn5   0/1     ContainerCreating   0          5m42s
pod/multus-admission-controller-nnkrx   0/1     ContainerCreating   0          5m42s
pod/multus-admission-controller-ztx78   0/1     ContainerCreating   0          5m42s
pod/multus-g2ckc                        1/1     Running             0          5m44s
pod/multus-sb7rf                        1/1     Running             0          5m44s

NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/multus-admission-controller   ClusterIP   172.30.166.59   <none>        443/TCP   5m44s

NAME                                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
daemonset.apps/multus                        6         6         6       6            6           kubernetes.io/os=linux            5m44s
daemonset.apps/multus-admission-controller   3         3         0       3            0           node-role.kubernetes.io/master=   5m42s
Glenns-MacBook-Pro-2:gw.lo gwest$ oc describe pod/multus-admission-controller-8kjn5
Name:               multus-admission-controller-8kjn5
Namespace:          openshift-multus
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               control-plane-0/192.168.1.201
Start Time:         Mon, 05 Aug 2019 16:29:17 -0500
Labels:             app=multus-admission-controller
                    component=network
                    controller-revision-hash=6f589d4f69
                    namespace=openshift-multus
                    openshift.io/component=network
                    pod-template-generation=1
                    type=infra
Annotations:        ovn: {"ip_address":"10.129.2.3/23", "mac_address":"0a:30:77:81:02:04", "gateway_ip": "10.129.2.1"}
Status:             Pending
IP:                 
Controlled By:      DaemonSet/multus-admission-controller
Containers:
  multus-admission-controller:
    Container ID:  
    Image:         registry.svc.ci.openshift.org/ocp/4.2-2019-08-05-184332@sha256:d141522c3c19f598a50765dcdbcd56f70c4c3ccae0004d5922908eebba7b6205
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/webhook
    Args:
      -bind-address=0.0.0.0
      -port=6443
      -tls-private-key-file=/etc/webhook/tls.key
      -tls-cert-file=/etc/webhook/tls.crt
      -alsologtostderr=true
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        10m
    Environment:  <none>
    Mounts:
      /etc/webhook from webhook-certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from multus-token-smw8t (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  webhook-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  multus-admission-controller-secret
    Optional:    false
  multus-token-smw8t:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  multus-token-smw8t
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason           Age                     From                      Message
  ----     ------           ----                    ----                      -------
  Normal   Scheduled        5m57s                   default-scheduler         Successfully assigned openshift-multus/multus-admission-controller-8kjn5 to control-plane-0
  Warning  NetworkNotReady  5m23s (x18 over 5m57s)  kubelet, control-plane-0  network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network
  Warning  FailedMount      5m23s (x7 over 5m56s)   kubelet, control-plane-0  MountVolume.SetUp failed for volume "webhook-certs" : secrets "multus-admission-controller-secret" not found
Glenns-MacBook-Pro-2:gw.lo gwest$

Comment 13 Glenn West 2019-08-05 22:27:58 UTC
EBUG OpenShift Installer unreleased-master-1508-g5e9cde9c162b313eed46ad964c9211f90e633681-dirty 
DEBUG Built from commit 5e9cde9c162b313eed46ad964c9211f90e633681 

Glenns-MacBook-Pro-2:gw.lo gwest$ oc get nodes
NAME              STATUS   ROLES    AGE    VERSION
control-plane-0   Ready    master   5m2s   v1.14.0+506b94723
control-plane-1   Ready    master   5m2s   v1.14.0+506b94723
control-plane-2   Ready    master   5m2s   v1.14.0+506b94723
worker-0          Ready    worker   5m9s   v1.14.0+506b94723
worker-1          Ready    worker   5m2s   v1.14.0+506b94723
worker-2          Ready    worker   5m2s   v1.14.0+506b94723
Glenns-MacBook-Pro-2:gw.lo gwest$ 

NAMESPACE                  NAME                                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
openshift-multus           daemonset.apps/multus                        6         6         6       6            6           kubernetes.io/os=linux            5m20s
openshift-multus           daemonset.apps/multus-admission-controller   3         3         0       3            0           node-role.kubernetes.io/master=   5m18s
openshift-ovn-kubernetes   daemonset.apps/ovnkube-node                  6         6         6       6            6           beta.kubernetes.io/os=linux       5m13s

Glenns-MacBook-Pro-2:gw.lo gwest$ oc project openshift-multus
Now using project "openshift-multus" on server "https://api.gw.lo:6443".
Glenns-MacBook-Pro-2:gw.lo gwest$ oc get all
NAME                                    READY   STATUS              RESTARTS   AGE
pod/multus-2mxtv                        1/1     Running             0          6m2s
pod/multus-54kwg                        1/1     Running             0          6m2s
pod/multus-72wnn                        1/1     Running             0          6m2s
pod/multus-admission-controller-9pqgs   0/1     ContainerCreating   0          6m
pod/multus-admission-controller-c2ng4   0/1     ContainerCreating   0          6m
pod/multus-admission-controller-cdqqx   0/1     ContainerCreating   0          6m
pod/multus-klpkt                        1/1     Running             0          6m2s
pod/multus-l5g92                        1/1     Running             0          6m2s
pod/multus-swtlt                        1/1     Running             0          6m2s
Glenns-MacBook-Pro-2:gw.lo gwest$ oc describe pod/multus-admission-controller-9pqgs
Name:               multus-admission-controller-9pqgs
Namespace:          openshift-multus
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               control-plane-0/192.168.1.201
Start Time:         Mon, 05 Aug 2019 17:17:06 -0500
Labels:             app=multus-admission-controller
                    component=network
                    controller-revision-hash=dcc8976bb
                    namespace=openshift-multus
                    openshift.io/component=network
                    pod-template-generation=1
                    type=infra
Annotations:        ovn: {"ip_address":"10.128.2.3/23", "mac_address":"6e:a6:fa:80:02:04", "gateway_ip": "10.128.2.1"}
Status:             Pending
IP:                 
Controlled By:      DaemonSet/multus-admission-controller
Containers:
  multus-admission-controller:
    Container ID:  
    Image:         registry.svc.ci.openshift.org/ocp/4.2-2019-08-05-191858@sha256:d141522c3c19f598a50765dcdbcd56f70c4c3ccae0004d5922908eebba7b6205
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/webhook
    Args:
      -bind-address=0.0.0.0
      -port=6443
      -tls-private-key-file=/etc/webhook/tls.key
      -tls-cert-file=/etc/webhook/tls.crt
      -alsologtostderr=true
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        10m
    Environment:  <none>
    Mounts:
      /etc/webhook from webhook-certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from multus-token-wgdkd (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  webhook-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  multus-admission-controller-secret
    Optional:    false
  multus-token-wgdkd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  multus-token-wgdkd
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason           Age                    From                      Message
  ----     ------           ----                   ----                      -------
  Normal   Scheduled        6m36s                  default-scheduler         Successfully assigned openshift-multus/multus-admission-controller-9pqgs to control-plane-0
  Warning  NetworkNotReady  6m2s (x18 over 6m36s)  kubelet, control-plane-0  network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network
  Warning  FailedMount      6m2s (x7 over 6m35s)   kubelet, control-plane-0  MountVolume.SetUp failed for volume "webhook-certs" : secrets "multus-admission-controller-secret" not found
  Warning  FailedMount      38s (x2 over 2m52s)    kubelet, control-plane-0  Unable to mount volumes for pod "multus-admission-controller-9pqgs_openshift-multus(c29b81f3-b7ce-11e9-a5d3-0050561f2525)": timeout expired waiting for volumes to attach or mount for pod "openshift-multus"/"multus-admission-controller-9pqgs". list of unmounted volumes=[webhook-certs]. list of unattached volumes=[webhook-certs multus-token-wgdkd]
Glenns-MacBook-Pro-2:gw.lo gwest$ 

Glenns-MacBook-Pro-2:gw.lo gwest$ oc logs pod/network-operator-cc6d95d6c-gnkvq
2019/08/05 22:17:02 Go Version: go1.10.8
2019/08/05 22:17:02 Go OS/Arch: linux/amd64
2019/08/05 22:17:02 operator-sdk Version: v0.4.1
2019/08/05 22:17:02 overriding kubernetes api to https://api-int.gw.lo:6443
2019/08/05 22:17:02 Registering Components.
2019/08/05 22:17:02 Configuring Controllers
2019/08/05 22:17:02 Starting the Cmd.
2019/08/05 22:17:03 Reconciling Network.config.openshift.io cluster
2019/08/05 22:17:03 reconciling (operator.openshift.io/v1, Kind=Network) /cluster
2019/08/05 22:17:03 does not exist, creating (operator.openshift.io/v1, Kind=Network) /cluster
2019/08/05 22:17:03 Reconciling Network.operator.openshift.io cluster
2019/08/05 22:17:03 Detected uplink MTU 1500
2019/08/05 22:17:03 Starting render phase
2019/08/05 22:17:03 successfully created (operator.openshift.io/v1, Kind=Network) /cluster
2019/08/05 22:17:03 Render phase done, rendered 25 objects
2019/08/05 22:17:03 reconciling (/v1, Kind=ConfigMap) openshift-network-operator/applied-cluster
2019/08/05 22:17:03 does not exist, creating (/v1, Kind=ConfigMap) openshift-network-operator/applied-cluster
2019/08/05 22:17:03 Reconciling update for openshift-service-ca from openshift-network-operator/cluster
2019/08/05 22:17:03 ConfigMap "openshift-service-ca" not found
2019/08/05 22:17:03 successfully created (/v1, Kind=ConfigMap) openshift-network-operator/applied-cluster
2019/08/05 22:17:03 reconciling (apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition) /network-attachment-definitions.k8s.cni.cncf.io
2019/08/05 22:17:03 does not exist, creating (apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition) /network-attachment-definitions.k8s.cni.cncf.io
2019/08/05 22:17:03 Created ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:03Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:03Z"
  status: "True"
  type: Upgradeable
2019/08/05 22:17:03 successfully created (apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition) /network-attachment-definitions.k8s.cni.cncf.io
2019/08/05 22:17:03 reconciling (/v1, Kind=Namespace) /openshift-multus
2019/08/05 22:17:03 does not exist, creating (/v1, Kind=Namespace) /openshift-multus
2019/08/05 22:17:03 successfully created (/v1, Kind=Namespace) /openshift-multus
2019/08/05 22:17:03 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /multus
2019/08/05 22:17:03 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /multus
2019/08/05 22:17:03 successfully created (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /multus
2019/08/05 22:17:03 reconciling (/v1, Kind=ServiceAccount) openshift-multus/multus
2019/08/05 22:17:03 does not exist, creating (/v1, Kind=ServiceAccount) openshift-multus/multus
2019/08/05 22:17:03 successfully created (/v1, Kind=ServiceAccount) openshift-multus/multus
2019/08/05 22:17:03 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /multus
2019/08/05 22:17:03 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /multus
2019/08/05 22:17:03 successfully created (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /multus
2019/08/05 22:17:03 reconciling (apps/v1, Kind=DaemonSet) openshift-multus/multus
2019/08/05 22:17:04 does not exist, creating (apps/v1, Kind=DaemonSet) openshift-multus/multus
2019/08/05 22:17:04 Reconciling update for openshift-service-ca from openshift-network-operator/cluster
2019/08/05 22:17:04 ConfigMap "openshift-service-ca" not found
2019/08/05 22:17:04 successfully created (apps/v1, Kind=DaemonSet) openshift-multus/multus
2019/08/05 22:17:04 reconciling (/v1, Kind=Service) openshift-multus/multus-admission-controller
2019/08/05 22:17:04 Reconciling update to openshift-multus/multus
2019/08/05 22:17:04 Error getting DaemonSet "openshift-multus/multus-admission-controller": DaemonSet.apps "multus-admission-controller" not found
2019/08/05 22:17:04 Error getting DaemonSet "openshift-ovn-kubernetes/ovnkube-node": DaemonSet.apps "ovnkube-node" not found
2019/08/05 22:17:04 Error getting Deployment "openshift-ovn-kubernetes/ovnkube-master": Deployment.apps "ovnkube-master" not found
2019/08/05 22:17:04 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
2019/08/05 22:17:04 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not yet scheduled on any nodes
    Waiting for DaemonSet "openshift-multus/multus-admission-controller" to be created
    Waiting for DaemonSet "openshift-ovn-kubernetes/ovnkube-node" to be created
    Waiting for Deployment "openshift-ovn-kubernetes/ovnkube-master" to be created
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:04 Reconciling update to openshift-multus/multus
2019/08/05 22:17:04 Error getting DaemonSet "openshift-multus/multus-admission-controller": DaemonSet.apps "multus-admission-controller" not found
2019/08/05 22:17:04 Error getting DaemonSet "openshift-ovn-kubernetes/ovnkube-node": DaemonSet.apps "ovnkube-node" not found
2019/08/05 22:17:04 Error getting Deployment "openshift-ovn-kubernetes/ovnkube-master": Deployment.apps "ovnkube-master" not found
2019/08/05 22:17:04 does not exist, creating (/v1, Kind=Service) openshift-multus/multus-admission-controller
2019/08/05 22:17:04 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" update is rolling out (4 out of 6 updated)
    Waiting for DaemonSet "openshift-multus/multus-admission-controller" to be created
    Waiting for DaemonSet "openshift-ovn-kubernetes/ovnkube-node" to be created
    Waiting for Deployment "openshift-ovn-kubernetes/ovnkube-master" to be created
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:04 Reconciling update to openshift-multus/multus
2019/08/05 22:17:04 Error getting DaemonSet "openshift-multus/multus-admission-controller": DaemonSet.apps "multus-admission-controller" not found
2019/08/05 22:17:04 Error getting DaemonSet "openshift-ovn-kubernetes/ovnkube-node": DaemonSet.apps "ovnkube-node" not found
2019/08/05 22:17:04 Error getting Deployment "openshift-ovn-kubernetes/ovnkube-master": Deployment.apps "ovnkube-master" not found
2019/08/05 22:17:04 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 6 nodes)
    Waiting for DaemonSet "openshift-multus/multus-admission-controller" to be created
    Waiting for DaemonSet "openshift-ovn-kubernetes/ovnkube-node" to be created
    Waiting for Deployment "openshift-ovn-kubernetes/ovnkube-master" to be created
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:04 successfully created (/v1, Kind=Service) openshift-multus/multus-admission-controller
2019/08/05 22:17:04 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /multus-admission-controller-webhook
2019/08/05 22:17:05 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /multus-admission-controller-webhook
2019/08/05 22:17:05 Reconciling update for openshift-service-ca from openshift-network-operator/cluster
2019/08/05 22:17:05 ConfigMap "openshift-service-ca" not found
2019/08/05 22:17:06 successfully created (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /multus-admission-controller-webhook
2019/08/05 22:17:06 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /multus-admission-controller-webhook
2019/08/05 22:17:06 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /multus-admission-controller-webhook
2019/08/05 22:17:06 successfully created (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /multus-admission-controller-webhook
2019/08/05 22:17:06 reconciling (admissionregistration.k8s.io/v1beta1, Kind=ValidatingWebhookConfiguration) /multus.openshift.io
2019/08/05 22:17:06 does not exist, creating (admissionregistration.k8s.io/v1beta1, Kind=ValidatingWebhookConfiguration) /multus.openshift.io
2019/08/05 22:17:06 successfully created (admissionregistration.k8s.io/v1beta1, Kind=ValidatingWebhookConfiguration) /multus.openshift.io
2019/08/05 22:17:06 reconciling (/v1, Kind=ConfigMap) openshift-network-operator/openshift-service-ca
2019/08/05 22:17:06 does not exist, creating (/v1, Kind=ConfigMap) openshift-network-operator/openshift-service-ca
2019/08/05 22:17:06 Reconciling update for openshift-service-ca from openshift-network-operator/cluster
2019/08/05 22:17:06 ConfigMap "openshift-service-ca" not found
2019/08/05 22:17:06 successfully created (/v1, Kind=ConfigMap) openshift-network-operator/openshift-service-ca
2019/08/05 22:17:06 reconciling (apps/v1, Kind=DaemonSet) openshift-multus/multus-admission-controller
2019/08/05 22:17:06 does not exist, creating (apps/v1, Kind=DaemonSet) openshift-multus/multus-admission-controller
2019/08/05 22:17:06 Reconciling update to openshift-multus/multus-admission-controller
2019/08/05 22:17:06 Error getting DaemonSet "openshift-ovn-kubernetes/ovnkube-node": DaemonSet.apps "ovnkube-node" not found
2019/08/05 22:17:06 Error getting Deployment "openshift-ovn-kubernetes/ovnkube-master": Deployment.apps "ovnkube-master" not found
2019/08/05 22:17:06 successfully created (apps/v1, Kind=DaemonSet) openshift-multus/multus-admission-controller
2019/08/05 22:17:06 reconciling (/v1, Kind=Namespace) /openshift-ovn-kubernetes
2019/08/05 22:17:06 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 6 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not yet scheduled on any nodes
    Waiting for DaemonSet "openshift-ovn-kubernetes/ovnkube-node" to be created
    Waiting for Deployment "openshift-ovn-kubernetes/ovnkube-master" to be created
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:06 Reconciling update to openshift-multus/multus-admission-controller
2019/08/05 22:17:06 Error getting DaemonSet "openshift-ovn-kubernetes/ovnkube-node": DaemonSet.apps "ovnkube-node" not found
2019/08/05 22:17:06 Error getting Deployment "openshift-ovn-kubernetes/ovnkube-master": Deployment.apps "ovnkube-master" not found
2019/08/05 22:17:07 does not exist, creating (/v1, Kind=Namespace) /openshift-ovn-kubernetes
2019/08/05 22:17:07 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 6 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" update is rolling out (1 out of 3 updated)
    Waiting for DaemonSet "openshift-ovn-kubernetes/ovnkube-node" to be created
    Waiting for Deployment "openshift-ovn-kubernetes/ovnkube-master" to be created
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:07 Reconciling update to openshift-multus/multus-admission-controller
2019/08/05 22:17:07 Error getting DaemonSet "openshift-ovn-kubernetes/ovnkube-node": DaemonSet.apps "ovnkube-node" not found
2019/08/05 22:17:07 Error getting Deployment "openshift-ovn-kubernetes/ovnkube-master": Deployment.apps "ovnkube-master" not found
2019/08/05 22:17:07 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 6 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    Waiting for DaemonSet "openshift-ovn-kubernetes/ovnkube-node" to be created
    Waiting for Deployment "openshift-ovn-kubernetes/ovnkube-master" to be created
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:07 Reconciling update for openshift-service-ca from openshift-network-operator/cluster
2019/08/05 22:17:07 successfully created (/v1, Kind=Namespace) /openshift-ovn-kubernetes
2019/08/05 22:17:07 reconciling (/v1, Kind=ServiceAccount) openshift-ovn-kubernetes/ovn-kubernetes-node
2019/08/05 22:17:07 does not exist, creating (/v1, Kind=ServiceAccount) openshift-ovn-kubernetes/ovn-kubernetes-node
2019/08/05 22:17:07 successfully created (/v1, Kind=ServiceAccount) openshift-ovn-kubernetes/ovn-kubernetes-node
2019/08/05 22:17:07 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /openshift-ovn-kubernetes-node
2019/08/05 22:17:07 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /openshift-ovn-kubernetes-node
2019/08/05 22:17:07 successfully created (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /openshift-ovn-kubernetes-node
2019/08/05 22:17:07 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /openshift-ovn-kubernetes-node
2019/08/05 22:17:08 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /openshift-ovn-kubernetes-node
2019/08/05 22:17:08 successfully created (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /openshift-ovn-kubernetes-node
2019/08/05 22:17:08 reconciling (/v1, Kind=ServiceAccount) openshift-ovn-kubernetes/ovn-kubernetes-controller
2019/08/05 22:17:08 does not exist, creating (/v1, Kind=ServiceAccount) openshift-ovn-kubernetes/ovn-kubernetes-controller
2019/08/05 22:17:08 successfully created (/v1, Kind=ServiceAccount) openshift-ovn-kubernetes/ovn-kubernetes-controller
2019/08/05 22:17:08 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /openshift-ovn-kubernetes-controller
2019/08/05 22:17:09 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /openshift-ovn-kubernetes-controller
2019/08/05 22:17:09 successfully created (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /openshift-ovn-kubernetes-controller
2019/08/05 22:17:09 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /openshift-ovn-kubernetes-controller
2019/08/05 22:17:09 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /openshift-ovn-kubernetes-controller
2019/08/05 22:17:09 successfully created (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /openshift-ovn-kubernetes-controller
2019/08/05 22:17:09 reconciling (rbac.authorization.k8s.io/v1, Kind=Role) openshift-ovn-kubernetes/openshift-ovn-kubernetes-sbdb
2019/08/05 22:17:09 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=Role) openshift-ovn-kubernetes/openshift-ovn-kubernetes-sbdb
2019/08/05 22:17:09 successfully created (rbac.authorization.k8s.io/v1, Kind=Role) openshift-ovn-kubernetes/openshift-ovn-kubernetes-sbdb
2019/08/05 22:17:09 reconciling (rbac.authorization.k8s.io/v1, Kind=RoleBinding) openshift-ovn-kubernetes/openshift-ovn-kubernetes-sbdb
2019/08/05 22:17:10 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=RoleBinding) openshift-ovn-kubernetes/openshift-ovn-kubernetes-sbdb
2019/08/05 22:17:10 successfully created (rbac.authorization.k8s.io/v1, Kind=RoleBinding) openshift-ovn-kubernetes/openshift-ovn-kubernetes-sbdb
2019/08/05 22:17:10 reconciling (/v1, Kind=ConfigMap) openshift-ovn-kubernetes/ovn-config
2019/08/05 22:17:10 does not exist, creating (/v1, Kind=ConfigMap) openshift-ovn-kubernetes/ovn-config
2019/08/05 22:17:10 successfully created (/v1, Kind=ConfigMap) openshift-ovn-kubernetes/ovn-config
2019/08/05 22:17:10 reconciling (/v1, Kind=Service) openshift-ovn-kubernetes/ovnkube-db
2019/08/05 22:17:10 does not exist, creating (/v1, Kind=Service) openshift-ovn-kubernetes/ovnkube-db
2019/08/05 22:17:11 successfully created (/v1, Kind=Service) openshift-ovn-kubernetes/ovnkube-db
2019/08/05 22:17:11 reconciling (apps/v1, Kind=Deployment) openshift-ovn-kubernetes/ovnkube-master
2019/08/05 22:17:11 does not exist, creating (apps/v1, Kind=Deployment) openshift-ovn-kubernetes/ovnkube-master
2019/08/05 22:17:11 Reconciling update to openshift-ovn-kubernetes/ovnkube-master
2019/08/05 22:17:11 Error getting DaemonSet "openshift-ovn-kubernetes/ovnkube-node": DaemonSet.apps "ovnkube-node" not found
2019/08/05 22:17:11 successfully created (apps/v1, Kind=Deployment) openshift-ovn-kubernetes/ovnkube-master
2019/08/05 22:17:11 reconciling (apps/v1, Kind=DaemonSet) openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:17:11 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 6 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    Waiting for DaemonSet "openshift-ovn-kubernetes/ovnkube-node" to be created
    Deployment "openshift-ovn-kubernetes/ovnkube-master" is not yet scheduled on any nodes
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:11 does not exist, creating (apps/v1, Kind=DaemonSet) openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:17:11 Reconciling update to openshift-ovn-kubernetes/ovnkube-master
2019/08/05 22:17:11 Error getting DaemonSet "openshift-ovn-kubernetes/ovnkube-node": DaemonSet.apps "ovnkube-node" not found
2019/08/05 22:17:12 Reconciling update to openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:17:12 successfully created (apps/v1, Kind=DaemonSet) openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:17:12 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 6 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not yet scheduled on any nodes
    Deployment "openshift-ovn-kubernetes/ovnkube-master" is not yet scheduled on any nodes
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:12 Reconciling update to openshift-ovn-kubernetes/ovnkube-master
2019/08/05 22:17:12 Failed to update ClusterOperator "network": Operation cannot be fulfilled on clusteroperators.config.openshift.io "network": the object has been modified; please apply your changes to the latest version and try again
2019/08/05 22:17:12 reconciling (config.openshift.io/v1, Kind=Network) /cluster
2019/08/05 22:17:13 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 6 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not yet scheduled on any nodes
    Deployment "openshift-ovn-kubernetes/ovnkube-master" is not available (awaiting 1 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:13 Reconciling Network.config.openshift.io cluster
2019/08/05 22:17:14 update was successful
2019/08/05 22:17:14 Reconciling update to openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:17:16 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 6 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" update is rolling out (3 out of 6 updated)
    Deployment "openshift-ovn-kubernetes/ovnkube-master" is not available (awaiting 1 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:16 Reconciling update to openshift-ovn-kubernetes/ovnkube-master
2019/08/05 22:17:17 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 6 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 6 nodes)
    Deployment "openshift-ovn-kubernetes/ovnkube-master" is not available (awaiting 1 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:17 Reconciling update to openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:17:29 Reconciling update to openshift-multus/multus
2019/08/05 22:17:29 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 5 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 6 nodes)
    Deployment "openshift-ovn-kubernetes/ovnkube-master" is not available (awaiting 1 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:30 Reconciling update to openshift-multus/multus
2019/08/05 22:17:30 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 4 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 6 nodes)
    Deployment "openshift-ovn-kubernetes/ovnkube-master" is not available (awaiting 1 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:32 Reconciling update to openshift-multus/multus
2019/08/05 22:17:32 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 3 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 6 nodes)
    Deployment "openshift-ovn-kubernetes/ovnkube-master" is not available (awaiting 1 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:17:57 Reconciling update to openshift-multus/multus
2019/08/05 22:17:57 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 2 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 6 nodes)
    Deployment "openshift-ovn-kubernetes/ovnkube-master" is not available (awaiting 1 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:18:02 Reconciling update to openshift-multus/multus
2019/08/05 22:18:02 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus" is not available (awaiting 1 nodes)
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 6 nodes)
    Deployment "openshift-ovn-kubernetes/ovnkube-master" is not available (awaiting 1 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:18:04 Reconciling update to openshift-multus/multus
2019/08/05 22:18:05 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 6 nodes)
    Deployment "openshift-ovn-kubernetes/ovnkube-master" is not available (awaiting 1 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:18:26 Reconciling update to openshift-ovn-kubernetes/ovnkube-master
2019/08/05 22:18:26 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 6 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:18:46 Reconciling update to openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:18:46 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 5 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:18:46 Reconciling update to openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:18:47 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 4 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:18:47 Reconciling update to openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:18:47 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 3 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:18:48 Reconciling update to openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:18:48 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 2 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:18:50 Reconciling update to openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:18:50 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: |-
    DaemonSet "openshift-multus/multus-admission-controller" is not available (awaiting 3 nodes)
    DaemonSet "openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 1 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:18:51 Reconciling update to openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:18:51 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "False"
  type: Degraded
- lastTransitionTime: "2019-08-05T22:17:04Z"
  status: "True"
  type: Upgradeable
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: DaemonSet "openshift-multus/multus-admission-controller" is not available
    (awaiting 3 nodes)
  reason: Deploying
  status: "True"
  type: Progressing
- lastTransitionTime: "2019-08-05T22:17:04Z"
  message: The network is starting up
  reason: Startup
  status: "False"
  type: Available
2019/08/05 22:22:04 Reconciling update to openshift-multus/multus
2019/08/05 22:22:06 Reconciling update to openshift-multus/multus-admission-controller
2019/08/05 22:22:11 Reconciling update to openshift-ovn-kubernetes/ovnkube-master
2019/08/05 22:22:12 Reconciling update to openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:22:14 Reconciling Network.operator.openshift.io cluster
2019/08/05 22:22:14 Starting render phase
2019/08/05 22:22:14 Render phase done, rendered 25 objects
2019/08/05 22:22:14 reconciling (/v1, Kind=ConfigMap) openshift-network-operator/applied-cluster
2019/08/05 22:22:14 update was successful
2019/08/05 22:22:14 reconciling (apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition) /network-attachment-definitions.k8s.cni.cncf.io
2019/08/05 22:22:14 update was successful
2019/08/05 22:22:14 reconciling (/v1, Kind=Namespace) /openshift-multus
2019/08/05 22:22:14 update was successful
2019/08/05 22:22:14 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /multus
2019/08/05 22:22:14 update was successful
2019/08/05 22:22:14 reconciling (/v1, Kind=ServiceAccount) openshift-multus/multus
2019/08/05 22:22:14 update was successful
2019/08/05 22:22:14 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /multus
2019/08/05 22:22:14 update was successful
2019/08/05 22:22:14 reconciling (apps/v1, Kind=DaemonSet) openshift-multus/multus
2019/08/05 22:22:14 update was successful
2019/08/05 22:22:14 reconciling (/v1, Kind=Service) openshift-multus/multus-admission-controller
2019/08/05 22:22:15 update was successful
2019/08/05 22:22:15 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /multus-admission-controller-webhook
2019/08/05 22:22:15 update was successful
2019/08/05 22:22:15 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /multus-admission-controller-webhook
2019/08/05 22:22:16 update was successful
2019/08/05 22:22:16 reconciling (admissionregistration.k8s.io/v1beta1, Kind=ValidatingWebhookConfiguration) /multus.openshift.io
2019/08/05 22:22:16 update was successful
2019/08/05 22:22:16 reconciling (/v1, Kind=ConfigMap) openshift-network-operator/openshift-service-ca
2019/08/05 22:22:16 reconciling (apps/v1, Kind=DaemonSet) openshift-multus/multus-admission-controller
2019/08/05 22:22:17 update was successful
2019/08/05 22:22:17 reconciling (/v1, Kind=Namespace) /openshift-ovn-kubernetes
2019/08/05 22:22:17 update was successful
2019/08/05 22:22:17 reconciling (/v1, Kind=ServiceAccount) openshift-ovn-kubernetes/ovn-kubernetes-node
2019/08/05 22:22:17 update was successful
2019/08/05 22:22:17 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /openshift-ovn-kubernetes-node
2019/08/05 22:22:18 update was successful
2019/08/05 22:22:18 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /openshift-ovn-kubernetes-node
2019/08/05 22:22:18 update was successful
2019/08/05 22:22:18 reconciling (/v1, Kind=ServiceAccount) openshift-ovn-kubernetes/ovn-kubernetes-controller
2019/08/05 22:22:19 update was successful
2019/08/05 22:22:19 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /openshift-ovn-kubernetes-controller
2019/08/05 22:22:19 update was successful
2019/08/05 22:22:19 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /openshift-ovn-kubernetes-controller
2019/08/05 22:22:19 update was successful
2019/08/05 22:22:19 reconciling (rbac.authorization.k8s.io/v1, Kind=Role) openshift-ovn-kubernetes/openshift-ovn-kubernetes-sbdb
2019/08/05 22:22:20 update was successful
2019/08/05 22:22:20 reconciling (rbac.authorization.k8s.io/v1, Kind=RoleBinding) openshift-ovn-kubernetes/openshift-ovn-kubernetes-sbdb
2019/08/05 22:22:20 update was successful
2019/08/05 22:22:20 reconciling (/v1, Kind=ConfigMap) openshift-ovn-kubernetes/ovn-config
2019/08/05 22:22:21 update was successful
2019/08/05 22:22:21 reconciling (/v1, Kind=Service) openshift-ovn-kubernetes/ovnkube-db
2019/08/05 22:22:21 update was successful
2019/08/05 22:22:21 reconciling (apps/v1, Kind=Deployment) openshift-ovn-kubernetes/ovnkube-master
2019/08/05 22:22:21 update was successful
2019/08/05 22:22:21 reconciling (apps/v1, Kind=DaemonSet) openshift-ovn-kubernetes/ovnkube-node
2019/08/05 22:22:22 update was successful
Glenns-MacBook-Pro-2:gw.lo gwest$     


Glenns-MacBook-Pro-2:gw.lo gwest$ oc project openshift-multus       
Now using project "openshift-multus" on server "https://api.gw.lo:6443".
Glenns-MacBook-Pro-2:gw.lo gwest$ oc get all
NAME                                    READY   STATUS              RESTARTS   AGE
pod/multus-2mxtv                        1/1     Running             0          9m55s
pod/multus-54kwg                        1/1     Running             0          9m55s
pod/multus-72wnn                        1/1     Running             0          9m55s
pod/multus-admission-controller-9pqgs   0/1     ContainerCreating   0          9m53s
pod/multus-admission-controller-c2ng4   0/1     ContainerCreating   0          9m53s
pod/multus-admission-controller-cdqqx   0/1     ContainerCreating   0          9m53s
pod/multus-klpkt                        1/1     Running             0          9m55s
pod/multus-l5g92                        1/1     Running             0          9m55s
pod/multus-swtlt                        1/1     Running             0          9m55s

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/multus-admission-controller   ClusterIP   172.30.232.174   <none>        443/TCP   9m55s

NAME                                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
daemonset.apps/multus                        6         6         6       6            6           kubernetes.io/os=linux            9m55s
daemonset.apps/multus-admission-controller   3         3         0       3            0           node-role.kubernetes.io/master=   9m53s
Glenns-MacBook-Pro-2:gw.lo gwest$ oc logs pod/multus-2mxtv
2019-08-05T22:17:56+0000 Generating Multus configuration file ...
2019-08-05T22:17:56+0000 Attemping to find master plugin configuration, attempt 0
2019-08-05T22:18:01+0000 Attemping to find master plugin configuration, attempt 5
2019-08-05T22:18:06+0000 Attemping to find master plugin configuration, attempt 10
2019-08-05T22:18:11+0000 Attemping to find master plugin configuration, attempt 15
2019-08-05T22:18:17+0000 Attemping to find master plugin configuration, attempt 20
2019-08-05T22:18:22+0000 Attemping to find master plugin configuration, attempt 25
2019-08-05T22:18:27+0000 Attemping to find master plugin configuration, attempt 30
2019-08-05T22:18:32+0000 Attemping to find master plugin configuration, attempt 35
2019-08-05T22:18:37+0000 Attemping to find master plugin configuration, attempt 40
2019-08-05T22:18:42+0000 Attemping to find master plugin configuration, attempt 45
2019-08-05T22:18:44+0000 Config file created @ /host/etc/cni/net.d/00-multus.conf
{ "name": "multus-cni-network", "type": "multus", "namespaceIsolation": true, "logLevel": "verbose", "kubeconfig": "/etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig", "delegates": [ {"cniVersion":"0.3.1","name":"ovn-kubernetes","type":"ovn-k8s-cni-overlay","ipam":{},"dns":{}} ] }
2019-08-05T22:18:44+0000 Entering sleep... (success)
Glenns-MacBook-Pro-2:gw.lo gwest$

Comment 14 zhaozhanqi 2019-08-06 01:47:01 UTC
hi, we have another similar bug https://bugzilla.redhat.com/show_bug.cgi?id=1729091 is tracing this issue

can we mark this bug as 'duplicate'.

Comment 15 Glenn West 2019-08-06 03:05:47 UTC
Least from looking at the logs, its a different issue.
I did look at the 1729091, and didn't see the same "markers" in this.

Comment 16 Glenn West 2019-08-06 03:12:49 UTC
After reviewing it another time, I don't mind trying a nightly that passes the dns bug.. So far I've tried every ci and nightly, and none have worked. Last working ovn based image was on 7-31.
As I would need to reopen, if its not duplicate, prefer not to mark it as duplicate.
We have several devs waiting on this to be "working" again.

Comment 18 Dan Winship 2019-08-13 14:36:07 UTC
the older bug ended up tracking this as well, since they can't verify that the dns bug was fixed until this bug is fixed too...

*** This bug has been marked as a duplicate of bug 1729091 ***


Note You need to log in before you can comment on or make changes to this bug.