Description of problem: Machine config operator is not finishing its initialization making install fails: [root@aux-server ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version False True 23m Unable to apply 4.2.0-0.nightly-2019-07-30-073644: the cluster operator machine-config is degraded [root@aux-server ~]# oc describe co machine-config Name: machine-config Namespace: Labels: <none> Annotations: <none> API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2019-07-30T09:29:50Z Generation: 1 Resource Version: 16727 Self Link: /apis/config.openshift.io/v1/clusteroperators/machine-config UID: 944946e2-b2ac-11e9-a0a6-0cc47ab5867c Spec: Status: Conditions: Last Transition Time: 2019-07-30T09:29:50Z Message: Cluster not available for 4.2.0-0.nightly-2019-07-30-073644 Status: False Type: Available Last Transition Time: 2019-07-30T09:29:50Z Message: Cluster is bootstrapping 4.2.0-0.nightly-2019-07-30-073644 Status: True Type: Progressing Last Transition Time: 2019-07-30T09:40:42Z Message: Failed to resync 4.2.0-0.nightly-2019-07-30-073644 because: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: configuration status for pool master is empty: pool is degraded because nodes fail with "3 nodes are reporting degraded status on sync": "Node master-01.morenod-0730.qe.devcluster.openshift.com is reporting: \"machineconfig.machineconfiguration.openshift.io \\\"rendered-master-182b0a5759e21f6df78e626bffaf8d9a\\\" not found\", Node master-00.morenod-0730.qe.devcluster.openshift.com is reporting: \"parsing booted osImageURL: parsing reference: \\\"<not pivoted>\\\": invalid reference format\", Node master-02.morenod-0730.qe.devcluster.openshift.com is reporting: \"machineconfig.machineconfiguration.openshift.io \\\"rendered-master-182b0a5759e21f6df78e626bffaf8d9a\\\" not found\"", retrying Reason: FailedToSync Status: True Type: Degraded Last Transition Time: 2019-07-30T09:40:42Z Reason: AsExpected Status: True Type: Upgradeable Extension: Last Sync Error: pool master has not progressed to latest configuration: configuration status for pool master is empty: pool is degraded because nodes fail with "3 nodes are reporting degraded status on sync": "Node master-01.morenod-0730.qe.devcluster.openshift.com is reporting: \"machineconfig.machineconfiguration.openshift.io \\\"rendered-master-182b0a5759e21f6df78e626bffaf8d9a\\\" not found\", Node master-00.morenod-0730.qe.devcluster.openshift.com is reporting: \"parsing booted osImageURL: parsing reference: \\\"<not pivoted>\\\": invalid reference format\", Node master-02.morenod-0730.qe.devcluster.openshift.com is reporting: \"machineconfig.machineconfiguration.openshift.io \\\"rendered-master-182b0a5759e21f6df78e626bffaf8d9a\\\" not found\"", retrying Related Objects: Group: Name: openshift-machine-config-operator Resource: namespaces Group: machineconfiguration.openshift.io Name: master Resource: machineconfigpools Group: machineconfiguration.openshift.io Name: worker Resource: machineconfigpools Group: machineconfiguration.openshift.io Name: cluster Resource: controllerconfigs Versions: Name: operator Version: 4.2.0-0.nightly-2019-07-30-073644 Events: <none> Version-Release number of the following components: Installer: 4.2.0-0.nightly-2019-07-30-073644 How reproducible: Steps to Reproduce: 1. Trigger installation on baremetal environemnt 2.Wait until cluster installation is completed 3.Check clusteroperator status Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated Expected results: Additional info: Please attach logs from ansible-playbook with the -vvv flag
How did you do this baremetal install? PXE? Where did you download the bootimage?
Or to say this more directly, you should be using the bootimage pinned by the installer, see also https://github.com/openshift/installer/issues/1399 And https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.2
Looks like CustomOrigin in rpm-ostree output is empty :/ if len(bootedDeployment.CustomOrigin) > 0 { The check above default to <not pivoted>
Can you provide MCD logs or must-gather please
https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/486
Verified on 4.2.0-0.nightly-2019-08-10-002649 + 42.80.20190809.0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922