Bug 1811144

Summary: Simultaneous changes to image.config.openshift.io/cluster results in multiple machineconfigs
Product: OpenShift Container Platform Reporter: Caden Marchese <cmarches>
Component: NodeAssignee: Urvashi Mohnani <umohnani>
Status: CLOSED DUPLICATE QA Contact: Sunil Choudhary <schoudha>
Severity: low Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: amurdaca, aos-bugs, jokerman
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-20 18:15:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Caden Marchese 2020-03-06 17:30:49 UTC
Description of problem:
Following the document at https://docs.openshift.com/container-platform/4.3/builds/setting-up-trusted-ca.html, and then https://docs.openshift.com/container-platform/4.3/openshift_images/image-configuration.html#images-configuration-insecure_image-configuration before the operator picks up the change occasionally results in multiple machineconfigs that conflict with each other.

Version-Release number of selected component (if applicable):
Openshift 4.3, AWS UPI, Private

How reproducible:
Somewhat

Steps to Reproduce:
1. Apply an update to add a trusted CA for a registry (https://docs.openshift.com/container-platform/4.3/builds/setting-up-trusted-ca.html)
2. Use 'oc patch' to apply the change 
3. Quickly also add a blocked registry or configuration change (https://docs.openshift.com/container-platform/4.3/openshift_images/image-configuration.html#images-configuration-insecure_image-configuration

Actual results:
Instability as a result of two conflicting configurations that overwrite each other. Note the timestamps in the master machine config pool:

NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
00-master                                                   25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d
00-worker                                                   25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d
01-master-container-runtime                                 25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d
01-master-kubelet                                           25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d
01-worker-container-runtime                                 25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d
01-worker-kubelet                                           25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d
99-master-f5d6922e-4244-4f3c-8fdb-8c97438d94ca-registries   25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d
99-master-ssh                                                                                          2.2.0             13d
99-worker-5bb1b481-477b-43fa-887c-86723aea3a2a-registries   25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d
99-worker-ssh                                                                                          2.2.0             13d
rendered-master-46a54ecdaeb9d1591ec7dfd736a66acc            25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             24m  <--- new
rendered-master-9e4c66aa9ed5f8986d052321957cfe4d            25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d  <--- old
rendered-master-a809ec58cc89d31d7fdb92eb05e24338            25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             57m  <--- new
rendered-master-d0b850ca7fd04c9c96e61ef65ce5df1d            25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d  <--- old
rendered-worker-3972ca77d7852f1899bf4ed3fde14ade            25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d
rendered-worker-97f9cea03aa9f88392ca97df618ced5d            25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             13d
rendered-worker-d64a8fcb5b4cf3a6d23acd5537c6fea1            25bb6aeb58135c38a667e849edf5244871be4992   2.2.0             57m

We have a master group MCP that is flipping between the old resource and new resource (see timestamps)

Expected results:
The machineconfig would normally be updated, rather than duplicated and replaced continually.

Additional info:
must-gather from customer's cluster as well as diff results between the two machineconfigs is available on request.

The workaround is to roll back to the machineconfig that predates the change to image.config.openshift.io/cluster.

Comment 5 Red Hat Bugzilla 2024-01-06 04:28:22 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days