Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1926722

Summary: MCP degraded after appling entitlement MachineConfig
Product: OpenShift Container Platform Reporter: Kevin Pouget <kpouget>
Component: Machine Config OperatorAssignee: MCO Team <team-mco>
Machine Config Operator sub component: Machine Config Operator QA Contact: Rio Liu <rioliu>
Status: CLOSED WONTFIX Docs Contact:
Severity: unspecified    
Priority: unspecified CC: aos-bugs, kgarriso, mkrejci, skumari
Version: 4.6   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-02 20:26:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kevin Pouget 2021-02-09 10:48:05 UTC 
Description of problem:

MCO failed to apply entitlement MachineConfig

After creating entitlement MachineConfig resources from this template:

https://gitlab.com/kpouget_psap/deploy-cluster/-/blob/master/utils/cluster-wide.entitlement.machineconfigs.yaml.template

the MCP got into a degraded state:

[stack@undercloud-0 tmp]$ oc describe mcp/worker
...
    Message:               Node titan-bdtdz-worker-0-6tss9 is reporting: "machineconfig.machineconfiguration.openshift.io \"rendered-worker-40057e6bcf6e59561ca8358ef499a897\" not found", Node
titan-bdtdz-worker-0-n59lg is reporting: "prepping update: machineconfig.machineconfiguration.openshift.io \"rendered-worker-40057e6bcf6e59561ca8358ef499a897\" not found", Node titan-bdtdz-worker-0-6hnpv is reporting: "prepping update: machineconfig.machineconfiguration.openshift.io \"rendered-worker-40057e6bcf6e59561ca8358ef499a897\" not found"
    Reason:                3 nodes are reporting degraded status on sync
...
    Source:
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   00-worker
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   01-worker-container-runtime
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   01-worker-kubelet
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   50-rhsm-conf
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-worker-generated-registries
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-worker-ssh
  Degraded Machine Count:     3
  Machine Count:              3
  Observed Generation:        14
  Ready Machine Count:        0
  Unavailable Machine Count:  2
  Updated Machine Count:      0

[stack@undercloud-0 tmp]$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-8dc467f8fc62364fecbcfa3898f71176   True      False      False      3              3                   3                     0                      4d19h
worker   rendered-worker-40057e6bcf6e59561ca8358ef499a897   False     True       True       3              0                   0                     3                      4d19h

[stack@undercloud-0 tmp]$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          fc2e69c4408d898b24760eea9e889f0673369e67   3.1.0             4d18h
00-worker                                          fc2e69c4408d898b24760eea9e889f0673369e67   3.1.0             4d18h
01-master-container-runtime                        fc2e69c4408d898b24760eea9e889f0673369e67   3.1.0             4d18h
01-master-kubelet                                  fc2e69c4408d898b24760eea9e889f0673369e67   3.1.0             4d18h
01-worker-container-runtime                        fc2e69c4408d898b24760eea9e889f0673369e67   3.1.0             4d18h
01-worker-kubelet                                  fc2e69c4408d898b24760eea9e889f0673369e67   3.1.0             4d18h
50-entitlement-key-pem                                                                        2.2.0             12h
50-entitlement-pem                                                                            2.2.0             12h
50-rhsm-conf                                                                                  2.2.0             12h
99-master-generated-registries                     fc2e69c4408d898b24760eea9e889f0673369e67   3.1.0             4d18h
99-master-ssh                                                                                 3.1.0             4d19h
99-worker-generated-registries                     fc2e69c4408d898b24760eea9e889f0673369e67   3.1.0             4d18h
99-worker-ssh                                                                                 3.1.0             4d19h
rendered-master-8dc467f8fc62364fecbcfa3898f71176   fc2e69c4408d898b24760eea9e889f0673369e67   3.1.0             4d18h
rendered-worker-53687c6ef65feaf200caebadf24bac59   fc2e69c4408d898b24760eea9e889f0673369e67   3.1.0             15m

Version-Release number of selected component (if applicable):


[stack@undercloud-0 tmp]$ oc version
Client Version: 4.6.16
Server Version: 4.6.16
Kubernetes Version: v1.19.0+e49167a

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Must-gather tarball:

http://file.emea.redhat.com/~kpouget/2021-02-09/must-gather.tgz