Bug 1868339

Hi Yunfei Jiang,

I am not able to currently pull a release image. Figuring out where to get the permissions.

Meantime, can you more elaborate on what's happening?

> there is a CredentialRequest in openshift-cluster-csi-drivers, but it is missing in the above oc comnand, this will cause installation failed when config CCO in manual mode, since user do not know the Secret for openshift-cluster-csi-drivers should be provided.

Are you saying that after running `oc adm release extract quay.io/openshift-release-dev/ocp-release:4.4.6-x86_64 --to ./release-image` (as described in https://github.com/openshift/cloud-credential-operator/blob/master/docs/mode-manual-creds.md), openshift-cluster-csi-drivers/openshift-cloud-credential-operator CredentialsRequest is completely missing from the extracted manifests?

*** Bug 1871713 has been marked as a duplicate of this bug. ***

A PR has been created to add the CredentialsRequests to the payload: https://github.com/openshift/cluster-storage-operator/pull/80

Once this is merged, subsequent PRs to CSI driver operator repos to NOT create those objects themselves are needed before we can verify this bug:

AWS EBS: https://github.com/openshift/aws-ebs-csi-driver-operator/blob/master/pkg/operator/starter.go#L70-L75
oVirt: https://github.com/openshift/ovirt-csi-driver-operator/blob/master/pkg/operator/starter.go#L86-L91
Manila: https://github.com/openshift/csi-driver-manila-operator/blob/master/pkg/operator/starter.go#L91-L96

Moving this back to ASSIGNED because other PRs to CSI driver operator repos are necessary to complete the transition.

Once all attached PRs are merged:

1) All CredentialsRequest should be applied by CVO
2) All CredentialsRequest should be available in the image payload
2) No CredentialsRequest will be applied by CSI driver operators (EBS, Manila and oVirt)

As part of the verification of this PR, please confirm if all of those CSI driver operators are installed and correctly.

PRs LGTM, new CR was added with correct permission:

version: 4.6.0-0.nightly-2020-09-03-005025

>> oc adm release extract $buildimage --cloud aws --credentials-requests

<--snip-->
---
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: aws-ebs-csi-driver-operator
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - action:
      - ec2:AttachVolume
      - ec2:CreateSnapshot
      - ec2:CreateTags
      - ec2:CreateVolume
      - ec2:DeleteSnapshot
      - ec2:DeleteTags
      - ec2:DeleteVolume
      - ec2:DescribeInstances
      - ec2:DescribeSnapshots
      - ec2:DescribeTags
      - ec2:DescribeVolumes
      - ec2:DescribeVolumesModifications
      - ec2:DetachVolume
      - ec2:ModifyVolume
      effect: Allow
      resource: '*'
  secretRef:
    name: ebs-cloud-credentials
    namespace: openshift-cluster-csi-drivers
<--snip-->

>> oc adm release extract registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-03-005025 --to ./release-image

0000_50_cluster-storage-operator_03_credentials_request_aws.yaml was extracted.

>> grep -l "apiVersion: cloudcredential.openshift.io" * | xargs cat

<--snip-->
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: aws-ebs-csi-driver-operator
  namespace: openshift-cloud-credential-operator
spec:
  secretRef:
    name: ebs-cloud-credentials
    namespace: openshift-cluster-csi-drivers
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - effect: Allow
      action:
      - ec2:AttachVolume
      - ec2:CreateSnapshot
      - ec2:CreateTags
      - ec2:CreateVolume
      - ec2:DeleteSnapshot
      - ec2:DeleteTags
      - ec2:DeleteVolume
      - ec2:DescribeInstances
      - ec2:DescribeSnapshots
      - ec2:DescribeTags
      - ec2:DescribeVolumes
      - ec2:DescribeVolumesModifications
      - ec2:DetachVolume
      - ec2:ModifyVolume
      #- ec2:*
      resource: "*"
<--snip-->

The 4.5 -> 4.6 upgrade process when CCO in manual mode is tracked by https://bugzilla.redhat.com/show_bug.cgi?id=1871713#c7

Hello Fabio,

These PRs make aws ebs csi driver operator and oVirt csi driver operator working well, but introduced a bug for manila csi driver operator.

The detail is:
1. The credentailsrequest created by manila csi driver operator,  its secret `manila-cloud-credentials` will be created in `openshift-manila-csi-driver` namespace.(https://github.com/openshift/csi-driver-manila-operator/pull/57/files line#8-9)
2. But the credentailsrequest created by CVO, its secret `manila-cloud-credentials` wil be created in `openshift-cluster-csi-drivers` namespace.(https://github.com/bertinatto/cluster-storage-operator/blob/2c521c2d7f18a804c1fc7360a9778dc3bc9dea99/manifests/03_credentials_request_openstack.yaml#L9)

So, PV dynamic provisioning does not work now.
 Warning  ProvisioningFailed    2m28s (x12 over 20m)  manila.csi.openstack.org_openstack-manila-csi-controllerplugin-6c65fb77f7-djspk_a8b9526c-e510-4771-8338-fe4b64ecd998  failed to provision volume with StorageClass "csi-manila-ceph": error getting secret csi-manila-secrets in namespace openshift-manila-csi-driver: secrets "csi-manila-secrets" not found

Check the opetor logs, it always pending to "Waiting for secret manila-cloud-credentials from cloud-credentials-operator"

So, I'll reassign this bug back.

verified with: 4.6.0-0.nightly-2020-09-03-224310

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196