Bug 2106051 - Unable to deploy acm-ice using latest SRO 4.11 build
Summary: Unable to deploy acm-ice using latest SRO 4.11 build
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Special Resource Operator
Version: 4.11
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.11.z
Assignee: Pablo Acevedo
QA Contact: Udi Kalifon
URL:
Whiteboard:
Depends On: 2107620
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-07-11 15:42 UTC by Bonnie Block
Modified: 2022-08-10 11:21 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2107620 (view as bug list)
Environment:
Last Closed: 2022-08-10 11:20:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift special-resource-operator pull 245 0 None open Bug 2106051: Update bundle 2022-07-15 14:33:22 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 11:21:09 UTC

Description Bonnie Block 2022-07-11 15:42:28 UTC
Description of problem:
Unable to deploy acm-ice using latest SRO build openshift-special-resource-operator.4.11.0-202207072008 

special-resource-controller-manager/manager log reports:
"error":"failed to get OCP versions: could not get version info from image 'quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda':

Version-Release number of selected component (if applicable):
4.11

How reproducible:
100%

Steps to Reproduce:
1.  Install SRO from latest brew build:
(https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2080106)

> reg=registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000
> dest_index=${reg}/sro/sro-index:v4.11
> opm index add --bundles registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-special-resource-operator-bundle@sha256:a833831d09bf798baa6cecbddc529a4971829d4507f33361d32340de446e812e --tag ${dest_index}
> podman push ${dest_index}
> oc adm catalog mirror ${dest_index} ${reg} -a /home/kni/mirror-combined-secret.json --manifests-only=true --icsp-scope registry --to-manifests=sro-manifests --filter-by-os=linux/amd64
> cp sro-manifests/mapping.txt sro-manifests/mapping-clone.txt
> sed -i 's|^registry.redhat.io/|brew.registry.redhat.io/|g' sro-manifests/mapping-clone.txt

# remove the registry.kni-qe-10.lab.eng.tlv2.redhat.com index image line from mapping-clone.txt
> vi sro-manifests/mapping-clone.txt

> oc image mirror -f sro-manifests/mapping-clone.txt -a /home/kni/mirror-combined-secret.json '--filter-by-os=.*' --keep-manifest-list --continue-on-error=true 
> cd sro-manifests
> oc apply -f catalogSource.yaml

# Create subscription for openshift-special-resource-operator: 

> vi sro-sub.yaml

>> apiVersion: operators.coreos.com/v1alpha1
>> kind: Subscription
>> metadata:
>>   name: openshift-special-resource-operator
>>   namespace: openshift-operators
>> spec:
>>   channel: "stable"
>>   installPlanApproval: Automatic
>>   name: openshift-special-resource-operator
>>   source: sro-index
>>   sourceNamespace: openshift-marketplace

> oc create -f sro-sub.yaml


2.  Deploy acm-ice:

# Download acm-ice SRO:
> git clone https://github.com/openshift/special-resource-operator.git
> cd special-resource-operator/charts/example

# Change policy remediationaction from enforce to inform and machineconfiguration.openshift.io/role from worker to master:
> perl -p -i -e 's/enforce/inform/g' acm-ice-0.0.1/templates/0002-policy.yaml
> perl -p -i -e 's/worker/master/g' acm-ice-0.0.1/templates/0002-policy.yaml

# Modify acm-ice-0.0.1/acm-ice.yaml:
> vi acm-ice-0.0.1/acm-ice.yaml

>> apiVersion: sro.openshift.io/v1beta1
>> kind: SpecialResourceModule
>> metadata:
>>   name: acm-ice
>> spec:
>>   namespace: acm-ice
>>   chart:
>>     name: acm-ice
>>     version: 0.0.1
>>     repository:
>>       name: acm-ice
>>       url: cm://acm-ice/acm-ice-chart
>>   set:
>>     kind: Values
>>     apiVersion: sro.openshift.io/v1beta1
>>     buildArgs:
>>       - name: DRIVER_VER
>>         value: "1.6.4"
>>     registry: registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000
>>   watch:
>>     - path: "$.metadata.labels.openshiftVersion"
>>       apiVersion: cluster.open-cluster-management.io/v1
>>       kind: ManagedCluster
>>       name: helix05


# Create helm chart and apply acm-ice.yaml:

> helm package acm-ice-0.0.1
> mkdir cm
> cp acm-ice-0.0.1.tgz cm/
> helm repo index cm --url=cm://acm-ice/acm-ice-chart
> oc new-project acm-ice
> oc create cm acm-ice-chart --from-file=cm/index.yaml --from-file=cm/acm-ice-0.0.1.tgz -n acm-ice
> oc apply -f acm-ice-0.0.1/acm-ice.yaml


Actual results:
No build or pod created.  Reconciler error in special-resource-controller-manager:

> $ oc logs -n openshift-operators special-resource-controller-manager-54c84488d9-8fc9k manager
>> {"level":"info","ts":1657553788.3302171,"logger":"controller.specialresourcemodule","msg":"Reconciling","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":""}
>> {"level":"info","ts":1657553788.3303373,"logger":"controller.specialresourcemodule","msg":"Reconciling watches","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":""}
>> {"level":"info","ts":1657553788.3303857,"logger":"controller.specialresourcemodule","msg":"Creating namespace","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":""}
>> {"level":"info","ts":1657553788.3376758,"logger":"controller.specialresourcemodule","msg":"Handling annotation specialresource.openshift.io/wait","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":"","objKind":"Namespace","objNamespace":"","objName":"acm-ice"}
>> {"level":"info","ts":1657553788.3407092,"logger":"controller.specialresourcemodule","msg":"Object is fully created, all wait conditions have been fulfilled","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":"","objKind":"Namespace","objNamespace":"","objName":"acm-ice"}
>> {"level":"info","ts":1657553798.0573053,"logger":"controller.specialresourcemodule","msg":"Version from regex","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":"","objectName":"helix05","element":"4.10.11"}
>> {"level":"error","ts":1657553798.0575635,"logger":"controller.specialresourcemodule","msg":"Reconciler error","reconciler group":"sro.openshift.io","reconciler kind":"SpecialResourceModule","name":"acm-ice","namespace":"","error":"failed to get OCP versions: could not get version info from image 'quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda': failed to get manifest's last layer for image 'quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda': failed to get layers digests of the image quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda: failed to get manifest stream from image quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda: failed to get crane manifest from image quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda: could not build crane options: could not find a pull source for \"quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda\": could not find registry for image \"quay.io/openshift-release-dev/ocp-release@sha256:0dc1a4b4d9ea7954987f63e506474a4f0dc55e5f1ea5c1f6f1179e2c09eaffda\": loading registries configuration \"/mnt/host/registries.conf\": open /mnt/host/registries.conf: no such file or directory","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}


Expected results:
Creation of acm-ice docker build

Additional info:
must-gather link:
https://drive.google.com/drive/folders/1NauJWL1ZfiFnvQOrbSjKd0d5_ZZ44qgY?usp=sharing

Comment 1 Brett Thurber 2022-07-11 15:48:42 UTC
@Udi, can you verify this?

Comment 2 Pablo Acevedo 2022-07-12 08:02:53 UTC
Looking at the error we can see this at the end:
loading registries configuration \"/mnt/host/registries.conf\": open /mnt/host/registries.conf: no such file or directory

This file is mounted from a host dir:
      containers:
        - [...]
          name: manager
          [...]
          volumeMounts:
          - name: cache-volume
            mountPath: /home/nonroot/.cache
          - name: host-registries-conf
            mountPath: /mnt/host/registries.conf
            readOnly: true
      volumes:
        - name: cache-volume
          emptyDir: {}
        - name: host-registries-conf
          hostPath:
            path: /etc/containers/registries.conf
            type: File

And from a freshly created 4.11 cluster, in one of the worker nodes:
[root@ip-10-0-150-205 /]# chroot /host
sh-4.4# cat /etc/containers/registries.conf
unqualified-search-registries = ['registry.access.redhat.com', 'docker.io']

Can you check if the file is there in any worker node?

Comment 3 Bonnie Block 2022-07-12 11:41:37 UTC
(In reply to Pablo Acevedo from comment #2)
> Looking at the error we can see this at the end:
> loading registries configuration \"/mnt/host/registries.conf\": open
> /mnt/host/registries.conf: no such file or directory
> 
> This file is mounted from a host dir:
>       containers:
>         - [...]
>           name: manager
>           [...]
>           volumeMounts:
>           - name: cache-volume
>             mountPath: /home/nonroot/.cache
>           - name: host-registries-conf
>             mountPath: /mnt/host/registries.conf
>             readOnly: true
>       volumes:
>         - name: cache-volume
>           emptyDir: {}
>         - name: host-registries-conf
>           hostPath:
>             path: /etc/containers/registries.conf
>             type: File
> 
> And from a freshly created 4.11 cluster, in one of the worker nodes:
> [root@ip-10-0-150-205 /]# chroot /host
> sh-4.4# cat /etc/containers/registries.conf
> unqualified-search-registries = ['registry.access.redhat.com', 'docker.io']
> 
> Can you check if the file is there in any worker node?

/etc/containers/registries.conf is present on each of the nodes:

[kni@registry example]$ oc get nodes
NAME                 STATUS   ROLES           AGE     VERSION
openshift-master-0   Ready    master,worker   4d14h   v1.23.5+3afdacb
openshift-master-1   Ready    master,worker   4d14h   v1.23.5+3afdacb
openshift-master-2   Ready    master,worker   4d14h   v1.23.5+3afdacb

[kni@registry example]$ ssh core@openshift-master-0
Red Hat Enterprise Linux CoreOS 410.84.202207051718-0
  Part of OpenShift 4.10, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.10/architecture/architecture-rhcos.html

---
Last login: Tue Jul 12 11:35:56 2022 from 10.46.62.1
[core@openshift-master-0 ~]$ cat /etc/containers/registries.conf
unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
short-name-mode = ""

[[registry]]
  prefix = ""
  location = "brew.registry.redhat.io"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005"

[[registry]]
  prefix = ""
  location = "quay.io/acm-d"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000/acm-d"

[[registry]]
  prefix = ""
  location = "quay.io/openshift-release-dev"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005/openshift-release-dev"

[[registry]]
  prefix = ""
  location = "quay.io/openshifttest"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005/openshifttest"

[[registry]]
  prefix = ""
  location = "quay.io/rh-nfv-int"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005/rh-nfv-int"

[[registry]]
  prefix = ""
  location = "registry-proxy.engineering.redhat.com"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000"

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005"

[[registry]]
  prefix = ""
  location = "registry.access.redhat.com/openshift4/ose-oauth-proxy"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000/openshift4/ose-oauth-proxy"

[[registry]]
  prefix = ""
  location = "registry.ci.openshift.org/ocp"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005/ocp"

[[registry]]
  prefix = ""
  location = "registry.connect.redhat.com"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000"

[[registry]]
  prefix = ""
  location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005"

[[registry]]
  prefix = ""
  location = "registry.redhat.io"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000"

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005"

[[registry]]
  prefix = ""
  location = "registry.redhat.io/multicluster-engine"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000/multicluster-engine"

[[registry]]
  prefix = ""
  location = "registry.redhat.io/openshift4/ose-kube-rbac-proxy"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000/openshift4/ose-kube-rbac-proxy"

[[registry]]
  prefix = ""
  location = "registry.redhat.io/rhacm2"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000/rhacm2"

[[registry]]
  prefix = ""
  location = "registry.stage.redhat.io"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5000"

  [[registry.mirror]]
    location = "registry.kni-qe-10.lab.eng.tlv2.redhat.com:5005"
[core@openshift-master-0 ~]$

Comment 7 Udi Kalifon 2022-07-29 07:24:18 UTC
Verified on 4.11-rc.5
I had to adapt the steps and change the name of the watched cluster, and once there was a match the build started running.

Comment 8 errata-xmlrpc 2022-08-10 11:20:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069


Note You need to log in before you can comment on or make changes to this bug.