I have replicated this on my AWS cluster, and I'm doing it again just now to confirm, my plan is to:

Install OCP 4.5.11
Update the machineconfig and confirm operation then upgrade to 4.5.20 again confirm operation then upgrade to 4.6.6 and see if it breaks like the last one did.

Previous test:

OCP 4.5.20 upgrade to OCP 4.6.6 then apply the machineconfig (which broke).

Regards,

Andy

signing the community and certified operator indexes is currently blocked by https://projects.engineering.redhat.com/browse/KODIAK-243 (in development, completing in Q1)

@Brian, In that case is the best course of action to advise the customer not to do this currently?

And if so why do we have this in the docs:

https://docs.openshift.com/container-platform/4.6/security/container_security/security-deploy.html#security-deploy-image-sources_security-deploy
https://access.redhat.com/verify-images-ocp4

Just so I can explain to my customer.

This seems like a docs oversight. There's not a quick fix here and it's something that we would probably need to advise the customer not to do until all of the catalogs shipped outside of the actual openshift payload themselves were signed. Even then, this would still prevent any operator images from working correctly if they were not also signed (which I assume will always be the case for the majority of community operators from the community catalog at the very least).

It works fine in OCP 4.5 because the catalogs were built at runtime by bootstrapping a container image that was included in the OCP payload. In 4.6 and later, we are now shipping the catalog separately from OCP itself. Those images come from separate sources and aren't part of the OCP build, and currently aren't signed.

While we should fix this (get the indexes signed, at least everything except community), i think we should also spawn an RFE to make the "signature required" feature more flexible, specifically by having allow/blocklists that allow you to bypass the signature requirement for specific registries/repositories/tags/SHAs.

MCO takes as input a containers policy file - not sure if this is the canonical doc location, but these docs describe the format: https://www.mankier.com/5/containers-policy.json

By my read of it, we should be able to add exceptions for the registry images, like:

```
{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ],
  "transports": {
    "docker": {
      "registry.access.redhat.com": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ],
      "registry.redhat.io/redhat/redhat-operator-index"": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ]
    },
    "docker-daemon": {
      "": [
        {
          "type": "insecureAcceptAnything"
        }
      ]
    }
  }
}
```

(this is untested and should be tailored for the customer)

If this approach works, we could probably close this out with an update to the linked docs: https://access.redhat.com/verify-images-ocp4

> We still want to get these images (the non-community catalog) signed though.

+1 

I'm not clear on what we plan to sign, but if it's everything from registry.redhat.io we can stop there.

If there's ever a world where an images are expected to be on registry.redhat.io but not signed (community operator images mirrored in?), there may be an RFE for tooling that can construct the policy file exceptions for users given an index.

I've created https://projects.engineering.redhat.com/browse/CLOUDDST-3881 to get the catalog images signed.

Thanks guys for all the great input on this topic, would be really good to get the images signed, as I am having to explain to the customer that he can used signed images in 4.5 but not in 4.6 currently (obviously that doesn't look great). Ben, I will track the JIRA ticket and this BZ using the customer case listed above.

Many thanks,

Andy

> as I am having to explain to the customer that he can used signed images in 4.5 but not in 4.6 currently (obviously that doesn't look great).

Did you attempt the mitigation I suggested that checks for signatures for images except for the signatures for the index images?

That should fix the imagepullbackoff issues for the indexes, though additional exceptions may need to be added for installed content from the indexes.

@Evan, this is the config we were testing with the customer (I have removed any customer registries)

{
    "default": [
        {
            "type": "reject"
        }
    ],
    "transports": {
        "atomic": {
            "quay.io": [
                {
                    "type": "insecureAcceptAnything"
                }
            ],
            "registry.access.redhat.com": [
                {
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release",
                    "keyType": "GPGKeys",
                    "type": "signedBy"
                }
            ],
            "registry.redhat.io": [
                {
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release",
                    "keyType": "GPGKeys",
                    "type": "signedBy"
                }
            ]
        },
        "docker": {
            "quay.io": [
                {
                    "type": "insecureAcceptAnything"
                }
            ],
            "registry.access.redhat.com": [
                {
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release",
                    "keyType": "GPGKeys",
                    "type": "signedBy"
                }
            ],
            "registry.redhat.io": [
                {
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release",
                    "keyType": "GPGKeys",
                    "type": "signedBy"
                }
            ]
        }
    }
}

This failed with the imagepull error

Could you try this and see if it addresses the issue?

{
    "default": [
        {
            "type": "reject"
        }
    ],
    "transports": {
        "atomic": {
            "quay.io": [
                {
                    "type": "insecureAcceptAnything"
                }
            ],
            "registry.access.redhat.com": [
                {
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release",
                    "keyType": "GPGKeys",
                    "type": "signedBy"
                }
            ],
            "registry.redhat.io/redhat/redhat-operator-index": [
                {
                    "type": "insecureAcceptAnything"
                }
            ],
            "registry.redhat.io/redhat/redhat-marketplace-index": [
                {
                    "type": "insecureAcceptAnything"
                }
            ],
            "registry.redhat.io/redhat/certified-operator-index": [
                {
                    "type": "insecureAcceptAnything"
                }
            ],
            "registry.redhat.io/redhat/community-operator-index": [
                {
                    "type": "insecureAcceptAnything"
                }
            ],
            "registry.redhat.io": [
                {
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release",
                    "keyType": "GPGKeys",
                    "type": "signedBy"
                }
            ]
        },
        "docker": {
            "quay.io": [
                {
                    "type": "insecureAcceptAnything"
                }
            ],
            "registry.access.redhat.com": [
                {
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release",
                    "keyType": "GPGKeys",
                    "type": "signedBy"
                }
            ],
            "registry.redhat.io": [
                {
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release",
                    "keyType": "GPGKeys",
                    "type": "signedBy"
                }
            ]
        }
    }
}

I just checked this in my lab. 

If the original policy looks the same as in https://access.redhat.com/verify-images-ocp4  

2 remarks about that article:
* for whatever reason that article sets default to insecureAcceptAnything and only forces validation for Red Hat images
* transport docker-daemon does not exist according to my man page:
~~~
[akaris@linux sigver2]$ man containers-policy.json | grep docker-daemon
[akaris@linux sigver2]$ 
~~~

That said, I reproduced the issue with that original policy:
~~~
cat <<'EOF' > policy.json
{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ],
  "transports": {
    "docker": {
      "registry.access.redhat.com": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ],
      "registry.redhat.io": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ]
    },
    "docker-daemon": {
      "": [
        {
          "type": "insecureAcceptAnything"
        }
      ]
    }
  }
}
EOF
~~~

Then this should be changed to:
~~~
cat <<'EOF' > policy.json
{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ],
  "transports": {
    "docker": {
      "registry.access.redhat.com": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ],
      "registry.redhat.io/redhat/redhat-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/redhat-marketplace-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/certified-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/community-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ]
    },
    "docker-daemon": {
      "": [
        {
          "type": "insecureAcceptAnything"
        }
      ]
    }
  }
}
EOF
~~~

Note that crio seems to pull via the docker: transport, not via the atomic: transport. So this means that the exceptions need to be set for that transport (as was done in comment #10, but the example from comment #18 did not work for me).

---------------------------------------------------------------

The workaround then is to manually connect to all worker nodes with oc debug node/<node name> and run:
~~~
oc debug node/...
chroot /host

cat <<'EOF' > /etc/containers/policy.json
{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ],
  "transports": {
    "docker": {
      "registry.access.redhat.com": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ],
      "registry.redhat.io/redhat/redhat-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/redhat-marketplace-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/certified-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/community-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ]
    },
    "docker-daemon": {
      "": [
        {
          "type": "insecureAcceptAnything"
        }
      ]
    }
  }
}
EOF
~~~

One can test immediately if this works by pulling the images manually with crictl:
~~~
crictl pull registry.redhat.io/redhat/certified-operator-index:v4.6
~~~

---------------------------

Then, one should also push the exception list via MachineConfig:
~~~
cat <<'EOF' > policy.json
{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ],
  "transports": {
    "docker": {
      "registry.access.redhat.com": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ],
      "registry.redhat.io/redhat/redhat-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/redhat-marketplace-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/certified-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/community-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ]
    },
    "docker-daemon": {
      "": [
        {
          "type": "insecureAcceptAnything"
        }
      ]
    }
  }
}
EOF

cat <<EOF > registry.access.redhat.com.yaml
docker:
     registry.access.redhat.com:
         sigstore: https://access.redhat.com/webassets/docker/content/sigstore
EOF

cat <<EOF > registry.redhat.io.yaml
docker:
     registry.redhat.io:
         sigstore: https://registry.redhat.io/containers/sigstore
EOF

export ARC_REG=$( cat registry.access.redhat.com.yaml | base64 -w0 )
export RIO_REG=$( cat registry.redhat.io.yaml | base64 -w0 )
export POLICY_CONFIG=$( cat policy.json | base64 -w0 )

cat > 51-worker-rh-registry-trust.yaml <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 51-worker-rh-registry-trust
spec:
  config:
    ignition:
      config: {}
      security:
        tls: {}
      timeouts: {}
      version: 2.2.0
    networkd: {}
    passwd: {}
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,${ARC_REG}
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/containers/registries.d/registry.access.redhat.com.yaml
      - contents:
          source: data:text/plain;charset=utf-8;base64,${RIO_REG}
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/containers/registries.d/registry.redhat.io.yaml
      - contents:
          source: data:text/plain;charset=utf-8;base64,${POLICY_CONFIG}
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/containers/policy.json
  osImageURL: ""
EOF

oc apply -f 51-worker-rh-registry-trust.yaml
~~~

(making non-confiential private comments public)

Moving this bz to medium given that there is a workaround that should not impact clusters and this BZ is only open for tracking purposes at this point -- the OLM component does not control the production of these images and just serves as an intermediary.

*** Bug 1929238 has been marked as a duplicate of this bug. ***

*** Bug 1939361 has been marked as a duplicate of this bug. ***

https://bugzilla.redhat.com/show_bug.cgi?id=1903632#c24 appears to outline the "correct" workaround steps.

Which as you say, are basically the same as https://access.redhat.com/verify-images-ocp4 and https://docs.openshift.com/container-platform/4.6/security/container_security/security-deploy.html#security-deploy-image-sources_security-deploy

Except that the policy.json content is slightly different, it needs to be what is contained in comment #24, namely the exceptions for the catalog images:

{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ],
  "transports": {
    "docker": {
      "registry.access.redhat.com": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ],
      "registry.redhat.io/redhat/redhat-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/redhat-marketplace-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/certified-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io/redhat/community-operator-index": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "registry.redhat.io": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ]
    },
    "docker-daemon": {
      "": [
        {
          "type": "insecureAcceptAnything"
        }
      ]
    }
  }
}


So can we just update https://access.redhat.com/verify-images-ocp4 w/ the modified policy.json and be sure that it calls out that this means the catalog images are not signature-verified (because they are not signed yet)?

Soon the redhat operator catalog will be signed and we can remove that exception.  The other 3 will be slightly trickier as they are going to be signed with a different key that doesn't exist on rhcos today.

[cloud-user@preserve-olm-env jian]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-09-14-200602   True        False         3h41m   Cluster version is 4.9.0-0.nightly-2021-09-14-200602

1, Update the container policy via the MachinConfig.
[cloud-user@preserve-olm-env jian]$ cat policy.json |base64 
ewoKICAgICJkZWZhdWx0IjogWwogICAgICAgIHsKICAgICAgICAgICAgInR5cGUiOiAiaW5zZWN1
cmVBY2NlcHRBbnl0aGluZyIKICAgICAgICB9CiAgICBdLAogICAgInRyYW5zcG9ydHMiOiB7CiAg
ICAgICAgImRvY2tlci1kYWVtb24iOiB7CiAgICAgICAgICAgICIiOiBbCiAgICAgICAgICAgICAg
ICB7CiAgICAgICAgICAgICAgICAgICAgInR5cGUiOiAiaW5zZWN1cmVBY2NlcHRBbnl0aGluZyIK
ICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgXQogICAgICAgIH0sCiAgICAgICAgImRvY2tl
ciI6IHsKICAgICAgICAgICAgInJlZ2lzdHJ5LnJlZGhhdC5pby9yZWRoYXQvY2VydGlmaWVkLW9w
ZXJhdG9yLWluZGV4IjogWwogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgICJ0
eXBlIjogInNpZ25lZEJ5IiwKICAgICAgICAgICAgICAgICAgICAia2V5VHlwZSI6ICJHUEdLZXlz
IiwKICAgICAgICAgICAgICAgICAgICAia2V5UGF0aCI6ICIvdXNyL3NoYXJlL29wZW5zaGlmdC9n
cGcta2V5cy9jb250YWluZXJpc3ZzaWduLmFzYyIKICAgICAgICAgICAgICAgIH0KICAgICAgICAg
ICAgXSwKICAgICAgICAgICAgInJlZ2lzdHJ5LnJlZGhhdC5pby9yZWRoYXQvY29tbXVuaXR5LW9w
ZXJhdG9yLWluZGV4IjogWwogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgICJ0
eXBlIjogInNpZ25lZEJ5IiwKICAgICAgICAgICAgICAgICAgICAia2V5VHlwZSI6ICJHUEdLZXlz
IiwKICAgICAgICAgICAgICAgICAgICAia2V5UGF0aCI6ICIvdXNyL3NoYXJlL29wZW5zaGlmdC9n
cGcta2V5cy9jb250YWluZXJpc3ZzaWduLmFzYyIKICAgICAgICAgICAgICAgIH0KICAgICAgICAg
ICAgXSwKICAgICAgICAgICAgInJlZ2lzdHJ5LnJlZGhhdC5pby9yZWRoYXQvcmVkaGF0LW1hcmtl
dHBsYWNlLWluZGV4IjogWwogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgICJ0
eXBlIjogInNpZ25lZEJ5IiwKICAgICAgICAgICAgICAgICAgICAia2V5VHlwZSI6ICJHUEdLZXlz
IiwKICAgICAgICAgICAgICAgICAgICAia2V5UGF0aCI6ICIvdXNyL3NoYXJlL29wZW5zaGlmdC9n
cGcta2V5cy9jb250YWluZXJpc3ZzaWduLmFzYyIKICAgICAgICAgICAgICAgIH0KICAgICAgICAg
ICAgXSwKICAgICAgICAgICAgInJlZ2lzdHJ5LnJlZGhhdC5pbyI6IFsKICAgICAgICAgICAgICAg
IHsKICAgICAgICAgICAgICAgICAgICAidHlwZSI6ICJzaWduZWRCeSIsCiAgICAgICAgICAgICAg
ICAgICAgImtleVR5cGUiOiAiR1BHS2V5cyIsCiAgICAgICAgICAgICAgICAgICAgImtleVBhdGgi
OiAiL2V0Yy9wa2kvcnBtLWdwZy9SUE0tR1BHLUtFWS1yZWRoYXQtcmVsZWFzZSIKICAgICAgICAg
ICAgICAgIH0KICAgICAgICAgICAgXQogICAgICAgIH0KICAgIH0KCn0K

[cloud-user@preserve-olm-env jian]$ cat machineconfig_update.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-zzz-worker-registry-trust
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,ewoKICAgICJkZWZhdWx0IjogWwogICAgICAgIHsKICAgICAgICAgICAgInR5cGUiOiAiaW5zZWN1cmVBY2NlcHRBbnl0aGluZyIKICAgICAgICB9CiAgICBdLAogICAgInRyYW5zcG9ydHMiOiB7CiAgICAgICAgImRvY2tlci1kYWVtb24iOiB7CiAgICAgICAgICAgICIiOiBbCiAgICAgICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAgICAgInR5cGUiOiAiaW5zZWN1cmVBY2NlcHRBbnl0aGluZyIKICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgXQogICAgICAgIH0sCiAgICAgICAgImRvY2tlciI6IHsKICAgICAgICAgICAgInJlZ2lzdHJ5LnJlZGhhdC5pby9yZWRoYXQvY2VydGlmaWVkLW9wZXJhdG9yLWluZGV4IjogWwogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgICJ0eXBlIjogInNpZ25lZEJ5IiwKICAgICAgICAgICAgICAgICAgICAia2V5VHlwZSI6ICJHUEdLZXlzIiwKICAgICAgICAgICAgICAgICAgICAia2V5UGF0aCI6ICIvdXNyL3NoYXJlL29wZW5zaGlmdC9ncGcta2V5cy9jb250YWluZXJpc3ZzaWduLmFzYyIKICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgXSwKICAgICAgICAgICAgInJlZ2lzdHJ5LnJlZGhhdC5pby9yZWRoYXQvY29tbXVuaXR5LW9wZXJhdG9yLWluZGV4IjogWwogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgICJ0eXBlIjogInNpZ25lZEJ5IiwKICAgICAgICAgICAgICAgICAgICAia2V5VHlwZSI6ICJHUEdLZXlzIiwKICAgICAgICAgICAgICAgICAgICAia2V5UGF0aCI6ICIvdXNyL3NoYXJlL29wZW5zaGlmdC9ncGcta2V5cy9jb250YWluZXJpc3ZzaWduLmFzYyIKICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgXSwKICAgICAgICAgICAgInJlZ2lzdHJ5LnJlZGhhdC5pby9yZWRoYXQvcmVkaGF0LW1hcmtldHBsYWNlLWluZGV4IjogWwogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgICJ0eXBlIjogInNpZ25lZEJ5IiwKICAgICAgICAgICAgICAgICAgICAia2V5VHlwZSI6ICJHUEdLZXlzIiwKICAgICAgICAgICAgICAgICAgICAia2V5UGF0aCI6ICIvdXNyL3NoYXJlL29wZW5zaGlmdC9ncGcta2V5cy9jb250YWluZXJpc3ZzaWduLmFzYyIKICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgXSwKICAgICAgICAgICAgInJlZ2lzdHJ5LnJlZGhhdC5pbyI6IFsKICAgICAgICAgICAgICAgIHsKICAgICAgICAgICAgICAgICAgICAidHlwZSI6ICJzaWduZWRCeSIsCiAgICAgICAgICAgICAgICAgICAgImtleVR5cGUiOiAiR1BHS2V5cyIsCiAgICAgICAgICAgICAgICAgICAgImtleVBhdGgiOiAiL2V0Yy9wa2kvcnBtLWdwZy9SUE0tR1BHLUtFWS1yZWRoYXQtcmVsZWFzZSIKICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgXQogICAgICAgIH0KICAgIH0KCn0K
        filesystem: root
        mode: 420
        path: /etc/containers/policy.json

[cloud-user@preserve-olm-env jian]$ oc create -f machineconfig_update.yaml 
machineconfig.machineconfiguration.openshift.io/99-zzz-worker-registry-trust created

[cloud-user@preserve-olm-env jian]$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             4h6m
00-worker                                          2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             4h6m
01-master-container-runtime                        2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             4h6m
01-master-kubelet                                  2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             4h6m
01-worker-container-runtime                        2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             4h6m
01-worker-kubelet                                  2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             4h6m
99-master-generated-registries                     2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             4h6m
99-master-ssh                                                                                 3.2.0             4h9m
99-worker-generated-registries                     2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             4h6m
99-worker-ssh                                                                                 3.2.0             4h9m
99-zzz-worker-registry-trust                                                                  3.1.0             126m
rendered-master-ade75d770442fd1258ee537395df5b35   2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             3h41m
rendered-master-b10059e938cd827d5755ad9e27f967d4   2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             4h6m
rendered-worker-168fcbfa9ad94a120f7e8059fdc776df   2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             125m
rendered-worker-56a2bd19ad9b6f7d406e1bf8f397558e   2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             3h41m
rendered-worker-7826ab548d5fa3264a9c2304564dcc65   2ec816a4aa741821e664fa512ab02f465926c0ab   3.2.0             4h6m
[cloud-user@preserve-olm-env jian]$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-ade75d770442fd1258ee537395df5b35   True      False      False      3              3                   3                     0                      4h8m
worker   rendered-worker-168fcbfa9ad94a120f7e8059fdc776df   True      False      False      3              3                   3                     0                      4h8m

2, Check if the policy works
[cloud-user@preserve-olm-env jian]$ oc get nodes
NAME                                         STATUS   ROLES    AGE     VERSION
ip-10-0-137-191.us-east-2.compute.internal   Ready    worker   3h57m   v1.22.0-rc.0+75ee307
ip-10-0-145-105.us-east-2.compute.internal   Ready    master   4h1m    v1.22.0-rc.0+75ee307
ip-10-0-172-210.us-east-2.compute.internal   Ready    worker   3h56m   v1.22.0-rc.0+75ee307
ip-10-0-173-43.us-east-2.compute.internal    Ready    master   4h3m    v1.22.0-rc.0+75ee307
ip-10-0-201-188.us-east-2.compute.internal   Ready    master   4h1m    v1.22.0-rc.0+75ee307
ip-10-0-213-186.us-east-2.compute.internal   Ready    worker   3h56m   v1.22.0-rc.0+75ee307
[cloud-user@preserve-olm-env jian]$ oc debug node/ip-10-0-137-191.us-east-2.compute.internal
Starting pod/ip-10-0-137-191us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.137.191
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# 
sh-4.4# 
sh-4.4# cat /etc/containers/policy.json
{

    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports": {
        "docker-daemon": {
            "": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]
        },
        "docker": {
            "registry.redhat.io/redhat/certified-operator-index": [
                {
                    "type": "signedBy",
                    "keyType": "GPGKeys",
                    "keyPath": "/usr/share/openshift/gpg-keys/containerisvsign.asc"
                }
            ],
            "registry.redhat.io/redhat/community-operator-index": [
                {
                    "type": "signedBy",
                    "keyType": "GPGKeys",
                    "keyPath": "/usr/share/openshift/gpg-keys/containerisvsign.asc"
                }
            ],
            "registry.redhat.io/redhat/redhat-marketplace-index": [
                {
                    "type": "signedBy",
                    "keyType": "GPGKeys",
                    "keyPath": "/usr/share/openshift/gpg-keys/containerisvsign.asc"
                }
            ],
            "registry.redhat.io": [
                {
                    "type": "signedBy",
                    "keyType": "GPGKeys",
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
                }
            ]
        }
    }

}

3, After a while, check the pods of openshift-marketplace project.

[cloud-user@preserve-olm-env jian]$ oc get pods -n openshift-marketplace
NAME                                    READY   STATUS              RESTARTS   AGE
certified-operators-5r7zd               0/1     Running             0          4s
certified-operators-s5g5n               1/1     Running             0          117m
community-operators-td4nq               1/1     Running             0          117m
marketplace-operator-6df6565cd6-ffphw   1/1     Running             0          4h3m
qe-app-registry-wmc4c                   1/1     Running             0          117m
qe-app-registry-zxwd4                   0/1     Running             0          4s
redhat-marketplace-lpc9b                1/1     Running             0          117m
redhat-operators-2fn2g                  0/1     ContainerCreating   0          4s
redhat-operators-95cnq                  1/1     Running             0          117m

[cloud-user@preserve-olm-env jian]$ oc get pods -n openshift-marketplace
NAME                                    READY   STATUS    RESTARTS   AGE
certified-operators-s5g5n               1/1     Running   0          126m
community-operators-td4nq               1/1     Running   0          126m
marketplace-operator-6df6565cd6-ffphw   1/1     Running   0          4h12m
qe-app-registry-wmc4c                   1/1     Running   0          126m
redhat-marketplace-lpc9b                1/1     Running   0          126m
redhat-operators-95cnq                  1/1     Running   0          126m

No crash, LGTM, verify it.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759