Bug 1695475

Created attachment 1551276 [details]
network_operator_log

Description of problem:
Trying to setup cluster with SRIOV enabled. The installation will report fail eventually with the network operator is still updating.

When checking the network operator after the installation finished (with fail), there is no version populated.


Version-Release number of selected component (if applicable):
4.0.0-0.nightly-2019-04-02-133735

How reproducible:
always

Steps to Reproduce:
1. Generate the manifests and update the cluster-network-03-config.yaml with following
# openshift-install create manifests
# cat manifests/cluster-network-03-config.yaml
apiVersion: "operator.openshift.io/v1"
kind: "Network"
metadata:
  name: "cluster"
spec:
  serviceNetwork:
  - "172.30.0.0/16"
  clusterNetwork:
  - cidr: "10.128.0.0/14"
    hostPrefix: 23
  defaultNetwork:
    type: OpenShiftSDN
    openshiftSDNConfig:
      mode: NetworkPolicy
  additionalNetworks:
    - type: Raw
      name: sriov-conf
      rawCNIConfig: '{
        "type": "sriov",
        "name": "sriov-network",
        "ipam": {
                "type": "host-local",
                "subnet": "10.11.11.0/24",
                "routes": [{
                        "dst": "0.0.0.0/0"
                }],
                "gateway": "10.11.11.1"
        }
      }'

2. Install the cluster
# openshift-install create cluster

3. Check the network operator
# oc get clusteroperator network

4. Check the cluster version
# oc get clusterversion

5. Check the operator pod log
# oc logs -f network-operator-d6c8c48b7-w8cm7 -n openshift-network-operator


Actual results:
Step2: The cluster installation will get failed eventually.
INFO Waiting up to 30m0s for the cluster at https://API_SERVER:6443 to initialize...
DEBUG Still waiting for the cluster to initialize: Working towards 4.0.0-0.nightly-2019-04-02-133735: 64% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.0.0-0.nightly-2019-04-02-133735: 88% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.0.0-0.nightly-2019-04-02-133735: 90% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.0.0-0.nightly-2019-04-02-133735: 91% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.0.0-0.nightly-2019-04-02-133735: 95% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.0.0-0.nightly-2019-04-02-133735: 97% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.0.0-0.nightly-2019-04-02-133735: 98% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.0.0-0.nightly-2019-04-02-133735: 98% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.0.0-0.nightly-2019-04-02-133735: 99% complete
DEBUG Still waiting for the cluster to initialize: Cluster operator network is still updating
FATAL failed to initialize the cluster: Cluster operator network is still updating: timed out waiting for the condition

Step3: There is no operator version populated.
# oc get clusteroperator network
NAME      VERSION   AVAILABLE   PROGRESSING   FAILING   SINCE
network             True        False         False     30m

Step4: The clusterversion shows that the cluster is not AVAILABLE.
# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-04-02-133735   False       True          25m     Unable to apply 4.0.0-0.nightly-2019-04-02-133735: the cluster operator network has not yet successfully rolled out

Step5: Full operator log attached.


Expected results:
Should be able to setup cluster successfully with the sriov enabled.

Additional info: