Description of problem: Unable to deploy acm-ice using latest SRO build openshift-special-resource-operator.4.11.0-202207072008 special-resource-controller-manager/manager log reports: "error":"failed to get OCP versions: could not get version info from image 'quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda': Version-Release number of selected component (if applicable): 4.11 How reproducible: 100% Steps to Reproduce: 1. Install SRO from latest brew build: (https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2080106) > reg=registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000 > dest_index=${reg}/sro/sro-index:v4.11 > opm index add --bundles registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-special-resource-operator-bundle@sha256:a833831d09bf798baa6cecbddc529a4971829d4507f33361d32340de446e812e --tag ${dest_index} > podman push ${dest_index} > oc adm catalog mirror ${dest_index} ${reg} -a /home/kni/mirror-combined-secret.json --manifests-only=true --icsp-scope registry --to-manifests=sro-manifests --filter-by-os=linux/amd64 > cp sro-manifests/mapping.txt sro-manifests/mapping-clone.txt > sed -i 's|^registry.redhat.io/|brew.registry.redhat.io/|g' sro-manifests/mapping-clone.txt # remove the registry.kni-qe-10.lab.eng.tlv2.redhat.com index image line from mapping-clone.txt > vi sro-manifests/mapping-clone.txt > oc image mirror -f sro-manifests/mapping-clone.txt -a /home/kni/mirror-combined-secret.json '--filter-by-os=.*' --keep-manifest-list --continue-on-error=true > cd sro-manifests > oc apply -f catalogSource.yaml # Create subscription for openshift-special-resource-operator: > vi sro-sub.yaml >> apiVersion: operators.coreos.com/v1alpha1 >> kind: Subscription >> metadata: >> name: openshift-special-resource-operator >> namespace: openshift-operators >> spec: >> channel: "stable" >> installPlanApproval: Automatic >> name: openshift-special-resource-operator >> source: sro-index >> sourceNamespace: openshift-marketplace > oc create -f sro-sub.yaml 2. Deploy acm-ice: # Download acm-ice SRO: > git clone https://github.com/openshift/special-resource-operator.git > cd special-resource-operator/charts/example # Change policy remediationaction from enforce to inform and machineconfiguration.openshift.io/role from worker to master: > perl -p -i -e 's/enforce/inform/g' acm-ice-0.0.1/templates/0002-policy.yaml > perl -p -i -e 's/worker/master/g' acm-ice-0.0.1/templates/0002-policy.yaml # Modify acm-ice-0.0.1/acm-ice.yaml: > vi acm-ice-0.0.1/acm-ice.yaml >> apiVersion: sro.openshift.io/v1beta1 >> kind: SpecialResourceModule >> metadata: >> name: acm-ice >> spec: >> namespace: acm-ice >> chart: >> name: acm-ice >> version: 0.0.1 >> repository: >> name: acm-ice >> url: cm://acm-ice/acm-ice-chart >> set: >> kind: Values >> apiVersion: sro.openshift.io/v1beta1 >> buildArgs: >> - name: DRIVER_VER >> value: "1.6.4" >> registry: registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000 >> watch: >> - path: "$.metadata.labels.openshiftVersion" >> apiVersion: cluster.open-cluster-management.io/v1 >> kind: ManagedCluster >> name: helix05 # Create helm chart and apply acm-ice.yaml: > helm package acm-ice-0.0.1 > mkdir cm > cp acm-ice-0.0.1.tgz cm/ > helm repo index cm --url=cm://acm-ice/acm-ice-chart > oc new-project acm-ice > oc create cm acm-ice-chart --from-file=cm/index.yaml --from-file=cm/acm-ice-0.0.1.tgz -n acm-ice > oc apply -f acm-ice-0.0.1/acm-ice.yaml Actual results: No build or pod created. Reconciler error in special-resource-controller-manager: > $ oc logs -n openshift-operators special-resource-controller-manager-54c84488d9-8fc9k manager >> {"level":"info","ts":1657553788.3302171,"logger":"controller.specialresourcemodule","msg":"Reconciling","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":""} >> {"level":"info","ts":1657553788.3303373,"logger":"controller.specialresourcemodule","msg":"Reconciling watches","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":""} >> {"level":"info","ts":1657553788.3303857,"logger":"controller.specialresourcemodule","msg":"Creating namespace","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":""} >> {"level":"info","ts":1657553788.3376758,"logger":"controller.specialresourcemodule","msg":"Handling annotation specialresource.openshift.io/wait","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":"","objKind":"Namespace","objNamespace":"","objName":"acm-ice"} >> {"level":"info","ts":1657553788.3407092,"logger":"controller.specialresourcemodule","msg":"Object is fully created, all wait conditions have been fulfilled","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":"","objKind":"Namespace","objNamespace":"","objName":"acm-ice"} >> {"level":"info","ts":1657553798.0573053,"logger":"controller.specialresourcemodule","msg":"Version from regex","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":"","objectName":"helix05","element":"4.10.11"} >> {"level":"error","ts":1657553798.0575635,"logger":"controller.specialresourcemodule","msg":"Reconciler error","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":"","error":"failed to get OCP versions: could not get version info from image 'quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda': failed to get manifest's last layer for image 'quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda': failed to get layers digests of the image quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda: failed to get manifest stream from image quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda: failed to get crane manifest from image quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda: could not build crane options: could not find a pull source for \"quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda\": could not find registry for image \"quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda\": loading registries configuration \"/mnt/host/registries.conf\": open /mnt/host/registries.conf: no such file or directory","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} Expected results: Creation of acm-ice docker build Additional info: must-gather link: https://drive.google.com/drive/folders/1NauJWL1ZfiFnvQOrbSjKd0d5_ZZ44qgY?usp=sharing
@Udi, can you verify this?
Looking at the error we can see this at the end: loading registries configuration \"/mnt/host/registries.conf\": open /mnt/host/registries.conf: no such file or directory This file is mounted from a host dir: containers: - [...] name: manager [...] volumeMounts: - name: cache-volume mountPath: /home/nonroot/.cache - name: host-registries-conf mountPath: /mnt/host/registries.conf readOnly: true volumes: - name: cache-volume emptyDir: {} - name: host-registries-conf hostPath: path: /etc/containers/registries.conf type: File And from a freshly created 4.11 cluster, in one of the worker nodes: [root@ip-10-0-150-205 /]# chroot /host sh-4.4# cat /etc/containers/registries.conf unqualified-search-registries = ['registry.access.redhat.com', 'docker.io'] Can you check if the file is there in any worker node?
(In reply to Pablo Acevedo from comment #2) > Looking at the error we can see this at the end: > loading registries configuration \"/mnt/host/registries.conf\": open > /mnt/host/registries.conf: no such file or directory > > This file is mounted from a host dir: > containers: > - [...] > name: manager > [...] > volumeMounts: > - name: cache-volume > mountPath: /home/nonroot/.cache > - name: host-registries-conf > mountPath: /mnt/host/registries.conf > readOnly: true > volumes: > - name: cache-volume > emptyDir: {} > - name: host-registries-conf > hostPath: > path: /etc/containers/registries.conf > type: File > > And from a freshly created 4.11 cluster, in one of the worker nodes: > [root@ip-10-0-150-205 /]# chroot /host > sh-4.4# cat /etc/containers/registries.conf > unqualified-search-registries = ['registry.access.redhat.com', 'docker.io'] > > Can you check if the file is there in any worker node? /etc/containers/registries.conf is present on each of the nodes: [kni@registry example]$ oc get nodes NAME STATUS ROLES AGE VERSION openshift-master-0 Ready master,worker 4d14h v1.23.5+3afdacb openshift-master-1 Ready master,worker 4d14h v1.23.5+3afdacb openshift-master-2 Ready master,worker 4d14h v1.23.5+3afdacb [kni@registry example]$ ssh core@openshift-master-0 Red Hat Enterprise Linux CoreOS 410.84.202207051718-0 Part of OpenShift 4.10, RHCOS is a Kubernetes native operating system managed by the Machine Config Operator (`clusteroperator/machine-config`). WARNING: Direct SSH access to machines is not recommended; instead, make configuration changes via `machineconfig` objects: https://docs.openshift.com/container-platform/4.10/architecture/architecture-rhcos.html --- Last login: Tue Jul 12 11:35:56 2022 from 10.46.62.1 [core@openshift-master-0 ~]$ cat /etc/containers/registries.conf unqualified-search-registries = ["registry.access.redhat.com", "docker.io"] short-name-mode = "" [[registry]] prefix = "" location = "brew.registry.redhat.io" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005" [[registry]] prefix = "" location = "quay.io/acm-d" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000/acm-d" [[registry]] prefix = "" location = "quay.io/openshift-release-dev" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005/openshift-release-dev" [[registry]] prefix = "" location = "quay.io/openshifttest" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005/openshifttest" [[registry]] prefix = "" location = "quay.io/rh-nfv-int" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005/rh-nfv-int" [[registry]] prefix = "" location = "registry-proxy.engineering.redhat.com" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000" [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005" [[registry]] prefix = "" location = "registry.access.redhat.com/openshift4/ose-oauth-proxy" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000/openshift4/ose-oauth-proxy" [[registry]] prefix = "" location = "registry.ci.openshift.org/ocp" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005/ocp" [[registry]] prefix = "" location = "registry.connect.redhat.com" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000" [[registry]] prefix = "" location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005" [[registry]] prefix = "" location = "registry.redhat.io" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000" [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005" [[registry]] prefix = "" location = "registry.redhat.io/multicluster-engine" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000/multicluster-engine" [[registry]] prefix = "" location = "registry.redhat.io/openshift4/ose-kube-rbac-proxy" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000/openshift4/ose-kube-rbac-proxy" [[registry]] prefix = "" location = "registry.redhat.io/rhacm2" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000/rhacm2" [[registry]] prefix = "" location = "registry.stage.redhat.io" mirror-by-digest-only = true [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000" [[registry.mirror]] location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005" [core@openshift-master-0 ~]$
Verified on 4.11-rc.5 I had to adapt the steps and change the name of the watched cluster, and once there was a match the build started running.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069