Description of problem:

Periodically machine configuration pools are updated after image.config.openshift.io/cluster and ImageContentSourcePolicy change	
If you compare /etc/containers/policy.json and /etc/containers/registries.conf entries aren't rendered in the same order.


Version-Release number of selected component (if applicable):
4.3

How reproducible:

Step 1: Patch image.config.openshift.io/cluster
# oc patch image.config.openshift.io/cluster -p '{"spec":{"registrySources":{"allowedRegistries":["my.rergistry.customer.fr","registry.redhat.io","quay.io"]}}}' --type='merge'
image.config.openshift.io/cluster patched
MCO applied config and rebooted the nodes

# oc get image.config.openshift.io/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Image
metadata:
  annotations:
    release.openshift.io/create-only: "true"
  creationTimestamp: "2020-03-25T14:31:19Z"
  generation: 5
  name: cluster
  resourceVersion: "10884240"
  selfLink: /apis/config.openshift.io/v1/images/cluster
  uid: 7b4fb3bf-f7c0-4a73-b98f-19096f570206
spec:
  registrySources:
    allowedRegistries:
    - my.rergistry.customer.fr
    - registry.redhat.io
    - quay.io

# ansible -i ~/ansible/ocp.yaml -m command -a "uptime" all
ocp4-lab1-w1.lab1.ocp.lab | CHANGED | rc=0 >> 08:37:39 up 13 min,  1 user,  load average: 0.13, 0.23, 0.22
ocp4-lab1-m0.lab1.ocp.lab | CHANGED | rc=0 >> 08:37:38 up 10 min,  1 user,  load average: 1.02, 1.04, 0.77
ocp4-lab1-m2.lab1.ocp.lab | CHANGED | rc=0 >> 08:37:38 up 14 min,  1 user,  load average: 0.85, 2.16, 1.88
ocp4-lab1-m1.lab1.ocp.lab | CHANGED | rc=0 >> 08:37:39 up 6 min,  1 user,  load average: 0.90, 0.96, 0.54
ocp4-lab1-w0.lab1.ocp.lab | CHANGED | rc=0 >> 08:37:37 up 15 min,  1 user,  load average: 2.18, 1.45, 0.92
No loop cluster is stable

# oc get nodes
NAME                        STATUS   ROLES    AGE   VERSION
ocp4-lab1-m0.lab1.ocp.lab   Ready    master   27d   v1.16.2
ocp4-lab1-m1.lab1.ocp.lab   Ready    master   27d   v1.16.2
ocp4-lab1-m2.lab1.ocp.lab   Ready    master   27d   v1.16.2
ocp4-lab1-w0.lab1.ocp.lab   Ready    worker   27d   v1.16.2
ocp4-lab1-w1.lab1.ocp.lab   Ready    worker   27d   v1.16.2

Step 2: create ImageContentSourcePolicy
# oc create -f - <<EOF
  apiVersion: operator.openshift.io/v1alpha1
  kind: ImageContentSourcePolicy
  metadata:
    name: ubi8repo
  spec:
   repositoryDigestMirrors:
    - mirrors:
      - example.io/example/ubi-minimal
      source: registry.access.redhat.com/ubi8/ubi-minimal
    - mirrors:
      - example.com/example/ubi-minimal
      source: registry.access.redhat.com/ubi8/ubi-minimal
EOF
imagecontentsourcepolicy.operator.openshift.io/ubi8repo created

# oc get ImageContentSourcePolicy -o yaml
apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1alpha1
  kind: ImageContentSourcePolicy
  metadata:
    creationTimestamp: "2020-04-22T09:08:39Z"
    generation: 1
    name: ubi8repo
    resourceVersion: "10901516"
    selfLink: /apis/operator.openshift.io/v1alpha1/imagecontentsourcepolicies/ubi8repo
    uid: 4307ca01-49bf-443b-a346-3db5c65564fa
  spec:
    repositoryDigestMirrors:
    - mirrors:
      - example.io/example/ubi-minimal
      source: registry.access.redhat.com/ubi8/ubi-minimal
    - mirrors:
      - example.com/example/ubi-minimal
      source: registry.access.redhat.com/ubi8/ubi-minimal
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

MCO applied config and reboot the nodes

# ansible -i ~/ansible/ocp.yaml -m command -a "uptime" all
ocp4-lab1-w1.lab1.ocp.lab | CHANGED | rc=0 >> 09:20:05 up 3 min,  1 user,  load average: 6.31, 1.53, 0.51
ocp4-lab1-m1.lab1.ocp.lab | CHANGED | rc=0 >> 09:20:05 up 4 min,  1 user,  load average: 3.35, 2.53, 1.18
ocp4-lab1-w0.lab1.ocp.lab | CHANGED | rc=0 >> 09:20:04 up 7 min,  1 user,  load average: 2.51, 2.88, 1.45
ocp4-lab1-m0.lab1.ocp.lab | CHANGED | rc=0 >> 09:20:05 up 8 min,  1 user,  load average: 9.80, 4.49, 2.11
ocp4-lab1-m2.lab1.ocp.lab | CHANGED | rc=0 >> 09:20:05 up 0 min,  1 user,  load average: 3.27, 0.81, 0.27


# oc debug node/ocp4-lab1-w0.lab1.ocp.lab
Starting pod/ocp4-lab1-w0lab1ocplab-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.110.110.30
If you don't see a command prompt, try pressing enter.
sh-4.2# cat /host/etc/containers/registries.conf 
unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]

[[registry]]
  prefix = ""
  location = "registry.access.redhat.com/ubi8/ubi-minimal"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "example.com/example/ubi-minimal"

  [[registry.mirror]]
    location = "example.io/example/ubi-minimal"

sh-4.2# cat /host/etc/containers/policy.json  
{"default":[{"type":"reject"}],"transports":{"atomic":{"my.rergistry.customer.fr":[{"type":"insecureAcceptAnything"}],"quay.io":[{"type":"insecureAcceptAnything"}],"registry.redhat.io":[{"type":"insecureAcceptAnything"}]},"docker":{"my.rergistry.customer.fr":[{"type":"insecureAcceptAnything"}],"quay.io":[{"type":"insecureAcceptAnything"}],"registry.redhat.io":[{"type":"insecureAcceptAnything"}]},"docker-daemon":{"":[{"type":"insecureAcceptAnything"}]}}}


After step 2 all nodes reboot periodically again and again. Here after an example based on node ocp4-lab1-w1.lab1.ocp.lab 
# oc describe node ocp4-lab1-w1.lab1.ocp.lab | grep reboot
  Warning  Rebooted 171m  kubelet, ocp4-lab1-w1.lab1.ocp.lab  Node ocp4-lab1-w1.lab1.ocp.lab has been rebooted, boot id: 4878cacb-da8f-4abc-8cf9-a2a49a79a559
  Warning  Rebooted 100m  kubelet, ocp4-lab1-w1.lab1.ocp.lab  Node ocp4-lab1-w1.lab1.ocp.lab has been rebooted, boot id: a8484103-e815-4612-898e-f786391278d1
  Warning  Rebooted 87m   kubelet, ocp4-lab1-w1.lab1.ocp.lab  Node ocp4-lab1-w1.lab1.ocp.lab has been rebooted, boot id: 94846862-b390-4229-91ec-6ca71f602314
  Warning  Rebooted 77m   kubelet, ocp4-lab1-w1.lab1.ocp.lab  Node ocp4-lab1-w1.lab1.ocp.lab has been rebooted, boot id: 91a3d6ea-f0af-4493-a644-31155f500680
  Warning  Rebooted 54m   kubelet, ocp4-lab1-w1.lab1.ocp.lab  Node ocp4-lab1-w1.lab1.ocp.lab has been rebooted, boot id: 1898a0bc-8ae6-4c30-8f64-5ad106bceff8

Actual results:
Nodes reboot in loop.
Looks like MCO overwrite image.config.openshift.io/cluster with ImageContentSourcePolicy and vice versa because rendering is not in the same order

Expected results:
MCO applies configuration only one time and not in loop

Additional info: