Description of problem: My customer has an issue when using both sigstore and whitelists together with the MCO We tried implementing a sigstore using https://access.redhat.com/verify-images-ocp4 and we do not lose our whitelist registry changes. But also cannot change policy.json to allow for image signature verification as described in the doc above. Looks like the MCO (or other operator?, had no luck finding the source code responsible for this) also dynamically creates a MachineConfig object for /etc/containers/policy.json based on the contents of the image.config.openshift.io/cluster resource (containing our whitelisting stuff), but with the always winning priority of 99. :) > oc get machineconfigpool/worker -o yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfigPool .... spec: configuration: name: rendered-worker-f40af94b2551f0c7613030f1f99c0af9 source: - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 00-worker - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 01-worker-container-runtime - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 01-worker-kubelet - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 51-worker-registry-trust <- contains our new policy.json - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 99-worker-862ae372-453a-4205-926d-abf8dea11270-registries <- contains generated policy.json - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 99-worker-mtu - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 99-worker-ssh - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: worker-chrony-configuration > oc get machineconfig 51-worker-registry-trust -o yaml | grep path: path: /etc/containers/registries.d/registry.redhat.io.yaml path: /etc/containers/registries.d/registry.access.redhat.com.yaml path: /etc/pki/<env>/openshift-virtual.<host>.pub path: /etc/containers/registries.d/openshift-virtual.<host>.yaml path: /etc/containers/policy.json > oc get machineconfig 99-worker-862ae372-453a-4205-926d-abf8dea11270-registries -o yaml | grep path: path: /etc/containers/policy.json Tested this by doing a manual change to image.config.openshift.io/cluster. The new entry to the spec.registrySources.allowedRegistries popped up in the generated MachineConfig and policy.json on the workers as a new registry. So looks like there is no way to win this... :) Ideally, we would like to set the sigstore and the whitelist through the same config option. It should be noted that this function is available in OCP 3.11, so why cant OCP 4.x do this? How reproducible: Everytime Steps to Reproduce: 1. 2. 3. Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated Expected results: Additional info: Please attach logs from ansible-playbook with the -vvv flag
This appears to related to configuring items via the MCO. Please file bugs against the component which you're dealing with.
Apologies Scott and thanks for correcting that for me. Regards, Andy
Ryan is on leave
Urvashi, seems like CU is trying to manually modify a file we manage with the ContainerRuntimeConfig controller https://github.com/openshift/machine-config-operator/blob/master/pkg/controller/container-runtime-config/container_runtime_config_controller.go Is there a way for them to resolve this (preferred)? Or should we just document the files that are managed by the ContainerRuntimeConfig controller are not directly changeable in the MachineConfig and provide some update for the sigstore docs (fallback)?
When you set allowedRegistries in the cluster wide Image CR, that edits the /etc/containers/policy.json file with the registries you are configuring as allowed. Looking at the sigstores steps, it is asking you to first manually create a policy.json and then create a Machine Config object for that. So the cu is using to different CR objects to make edits to the same file and the Image CR one is winning since it has a higher number. The solution here is to remove any allowed/blocked registries from the Image CR and to add it manually to the policy.json file being created to be used by MC as stated in the sigstores docs. You can look at the contents of /etc/containers/policy.json right now with the Image CR options set and copy/paste that to the policy.json file you are creating with the sigstores steps. Another way to pick up the sigstore policy.json is to rename the MC to have a higher number than 99, which will override the generated policy.json from 99-<hash>-registries so try 101-worker-registry-trust instead of 51-worker-registry-trust. We should probably document that if you are using sigstores, use a number higher than 99 for the MC name, or don't use the cluster-wide Image CR to modify allowed/blocked registries.
I am not entirely sure if all the components that use the Image CR also use sigstore. @bparees, @adambkaplan, and @dmage do you know if it would be possible to extend the Image CR to support sigstore?
I think we need a note somewhere in these two sections: https://docs.openshift.com/container-platform/4.11/security/container_security/security-container-signature.html#containers-signature-verify-enable_security-container-signature (basically a slightly updated version of https://access.redhat.com/verify-images-ocp4) https://docs.openshift.com/container-platform/4.11/openshift_images/image-configuration.html#images-configuration-file_image-configuration warning that changes to the image.config.openshift.io/cluster resource (as described in the first link) may overwrite/supersede any changes they may try to make to /etc/containers/policy.json directly (as described in the second link). I think the issue is with registrySources.allowedRegistries and registrySources.blockedRegistries specifically. According to https://issues.redhat.com/browse/RFE-1601 this conflict should eventually be fixed by https://issues.redhat.com/browse/OCPNODE-538
Thanks Andy. Will wait to hear from you.