According to the document https://github.com/openshift/cloud-credential-operator/blob/master/docs/mode-manual-creds.md all required CredentialRequest for config CCO in manual mode should be extracted by command `oc adm release extract`. there is a CredentialRequest in openshift-cluster-csi-drivers, but it is missing in the above oc comnand, this will cause installation failed when config CCO in manual mode, since user do not know the Secret for openshift-cluster-csi-drivers should be provided. After provide Secret for openshift-cluster-csi-drivers, the cluster could be installed successfully. >> errors if do not provide Secret for openshift-cluster-csi-drivers: level=info msg="Cluster operator storage Progressing is True with AWSEBSCSIDriverOperatorCR_AWSEBSDriverController_AsExpected: AWSEBSCSIDriverOperatorCRProgressing: AWSEBSDriverControllerProgressing: Waiting for Deployment to deploy controller pods" level=info msg="Cluster operator storage Available is False with AWSEBSCSIDriverOperatorCR_AWSEBSDriverController_AsExpected: AWSEBSCSIDriverOperatorCRAvailable: AWSEBSDriverControllerAvailable: Waiting for Deployment to deploy the CSI Controller Service" level=fatal msg="failed to initialize the cluster: Cluster operator storage has not yet reported success" >> CredentialRequest for openshift-cluster-csi-drivers: --- oc describe CredentialsRequest -n openshift-cloud-credential-operator openshift-cluster-csi-drivers Name: openshift-cluster-csi-drivers Namespace: openshift-cloud-credential-operator Labels: <none> Annotations: operator.openshift.io/spec-hash: ccbc9a71628daecd6066ff25e09d562be0a5f649dbbe44fb3c10ba39fd3616cf API Version: cloudcredential.openshift.io/v1 Kind: CredentialsRequest Metadata: Creation Timestamp: 2020-08-12T02:50:07Z Generation: 1 Managed Fields: API Version: cloudcredential.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:operator.openshift.io/spec-hash: f:spec: .: f:providerSpec: .: f:apiVersion: f:kind: f:statementEntries: f:secretRef: .: f:name: f:namespace: Manager: aws-ebs-csi-driver-operator Operation: Update Time: 2020-08-12T02:50:07Z Resource Version: 28521 Self Link: /apis/cloudcredential.openshift.io/v1/namespaces/openshift-cloud-credential-operator/credentialsrequests/openshift-cluster-csi-drivers UID: f58b58bd-896f-4449-8242-b605c61ef512 Spec: Provider Spec: API Version: cloudcredential.openshift.io/v1 Kind: AWSProviderSpec Statement Entries: Action: ec2:AttachVolume ec2:CreateSnapshot ec2:CreateTags ec2:CreateVolume ec2:DeleteSnapshot ec2:DeleteTags ec2:DeleteVolume ec2:DescribeInstances ec2:DescribeSnapshots ec2:DescribeTags ec2:DescribeVolumes ec2:DescribeVolumesModifications ec2:DetachVolume ec2:ModifyVolume Effect: Allow Resource: * Secret Ref: Name: aws-cloud-credentials Namespace: openshift-cluster-csi-drivers Events: <none> Version-Release number of the following components: 4.6.0-0.nightly-2020-08-10-180431 How reproducible: Always Steps to Reproduce: 1. Create install-config 2. Config CCO in manual mode, refer to https://github.com/openshift/cloud-credential-operator/blob/master/docs/mode-manual-creds.md 2. Create cluster Actual results: create cluster failed: level=info msg="Cluster operator storage Progressing is True with AWSEBSCSIDriverOperatorCR_AWSEBSDriverController_AsExpected: AWSEBSCSIDriverOperatorCRProgressing: AWSEBSDriverControllerProgressing: Waiting for Deployment to deploy controller pods" level=info msg="Cluster operator storage Available is False with AWSEBSCSIDriverOperatorCR_AWSEBSDriverController_AsExpected: AWSEBSCSIDriverOperatorCRAvailable: AWSEBSDriverControllerAvailable: Waiting for Deployment to deploy the CSI Controller Service" level=fatal msg="failed to initialize the cluster: Cluster operator storage has not yet reported success" Expected results: create cluster succesfully Additional info:
Hi Yunfei Jiang, I am not able to currently pull a release image. Figuring out where to get the permissions. Meantime, can you more elaborate on what's happening? > there is a CredentialRequest in openshift-cluster-csi-drivers, but it is missing in the above oc comnand, this will cause installation failed when config CCO in manual mode, since user do not know the Secret for openshift-cluster-csi-drivers should be provided. Are you saying that after running `oc adm release extract quay.io/openshift-release-dev/ocp-release:4.4.6-x86_64 --to ./release-image` (as described in https://github.com/openshift/cloud-credential-operator/blob/master/docs/mode-manual-creds.md), openshift-cluster-csi-drivers/openshift-cloud-credential-operator CredentialsRequest is completely missing from the extracted manifests?
*** Bug 1871713 has been marked as a duplicate of this bug. ***
A PR has been created to add the CredentialsRequests to the payload: https://github.com/openshift/cluster-storage-operator/pull/80 Once this is merged, subsequent PRs to CSI driver operator repos to NOT create those objects themselves are needed before we can verify this bug: AWS EBS: https://github.com/openshift/aws-ebs-csi-driver-operator/blob/master/pkg/operator/starter.go#L70-L75 oVirt: https://github.com/openshift/ovirt-csi-driver-operator/blob/master/pkg/operator/starter.go#L86-L91 Manila: https://github.com/openshift/csi-driver-manila-operator/blob/master/pkg/operator/starter.go#L91-L96
Moving this back to ASSIGNED because other PRs to CSI driver operator repos are necessary to complete the transition.
Once all attached PRs are merged: 1) All CredentialsRequest should be applied by CVO 2) All CredentialsRequest should be available in the image payload 2) No CredentialsRequest will be applied by CSI driver operators (EBS, Manila and oVirt) As part of the verification of this PR, please confirm if all of those CSI driver operators are installed and correctly.
PRs LGTM, new CR was added with correct permission: version: 4.6.0-0.nightly-2020-09-03-005025 >> oc adm release extract $buildimage --cloud aws --credentials-requests <--snip--> --- apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: aws-ebs-csi-driver-operator namespace: openshift-cloud-credential-operator spec: providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - action: - ec2:AttachVolume - ec2:CreateSnapshot - ec2:CreateTags - ec2:CreateVolume - ec2:DeleteSnapshot - ec2:DeleteTags - ec2:DeleteVolume - ec2:DescribeInstances - ec2:DescribeSnapshots - ec2:DescribeTags - ec2:DescribeVolumes - ec2:DescribeVolumesModifications - ec2:DetachVolume - ec2:ModifyVolume effect: Allow resource: '*' secretRef: name: ebs-cloud-credentials namespace: openshift-cluster-csi-drivers <--snip--> >> oc adm release extract registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-03-005025 --to ./release-image 0000_50_cluster-storage-operator_03_credentials_request_aws.yaml was extracted. >> grep -l "apiVersion: cloudcredential.openshift.io" * | xargs cat <--snip--> apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: aws-ebs-csi-driver-operator namespace: openshift-cloud-credential-operator spec: secretRef: name: ebs-cloud-credentials namespace: openshift-cluster-csi-drivers providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - effect: Allow action: - ec2:AttachVolume - ec2:CreateSnapshot - ec2:CreateTags - ec2:CreateVolume - ec2:DeleteSnapshot - ec2:DeleteTags - ec2:DeleteVolume - ec2:DescribeInstances - ec2:DescribeSnapshots - ec2:DescribeTags - ec2:DescribeVolumes - ec2:DescribeVolumesModifications - ec2:DetachVolume - ec2:ModifyVolume #- ec2:* resource: "*" <--snip--> The 4.5 -> 4.6 upgrade process when CCO in manual mode is tracked by https://bugzilla.redhat.com/show_bug.cgi?id=1871713#c7
Hello Fabio, These PRs make aws ebs csi driver operator and oVirt csi driver operator working well, but introduced a bug for manila csi driver operator. The detail is: 1. The credentailsrequest created by manila csi driver operator, its secret `manila-cloud-credentials` will be created in `openshift-manila-csi-driver` namespace.(https://github.com/openshift/csi-driver-manila-operator/pull/57/files line#8-9) 2. But the credentailsrequest created by CVO, its secret `manila-cloud-credentials` wil be created in `openshift-cluster-csi-drivers` namespace.(https://github.com/bertinatto/cluster-storage-operator/blob/2c521c2d7f18a804c1fc7360a9778dc3bc9dea99/manifests/03_credentials_request_openstack.yaml#L9) So, PV dynamic provisioning does not work now. Warning ProvisioningFailed 2m28s (x12 over 20m) manila.csi.openstack.org_openstack-manila-csi-controllerplugin-6c65fb77f7-djspk_a8b9526c-e510-4771-8338-fe4b64ecd998 failed to provision volume with StorageClass "csi-manila-ceph": error getting secret csi-manila-secrets in namespace openshift-manila-csi-driver: secrets "csi-manila-secrets" not found Check the opetor logs, it always pending to "Waiting for secret manila-cloud-credentials from cloud-credentials-operator" So, I'll reassign this bug back.
verified with: 4.6.0-0.nightly-2020-09-03-224310
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196