Bug 1868339 - CredentialsRequest for openshift-cluster-csi-drivers is missing in `oc adm release extract` command
Summary: CredentialsRequest for openshift-cluster-csi-drivers is missing in `oc adm re...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.6.0
Assignee: Fabio Bertinatto
QA Contact: Qin Ping
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-12 10:49 UTC by Yunfei Jiang
Modified: 2020-10-27 16:28 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:28:06 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift aws-ebs-csi-driver-operator pull 86 0 None closed Bug 1868339: Don't create CredentialsRequest in aws-ebs-csi-driver-operator 2020-10-07 10:10:32 UTC
Github openshift cluster-storage-operator pull 80 0 None closed Bug 1868339: Move CredentialsRequest creation to CSO 2020-10-07 10:10:32 UTC
Github openshift cluster-storage-operator pull 84 0 None closed Bug 1868339: Rename Manila CredentialsRequest to a name similar to the other ones 2020-10-07 10:10:31 UTC
Github openshift csi-driver-manila-operator pull 57 0 None closed Bug 1868339: Don't create CredentialsRequest in Manila CSI Driver Operator 2020-10-07 10:10:31 UTC
Github openshift csi-driver-manila-operator pull 58 0 None closed Bug 1868339: Use the CredentialsRequest created by CVO 2020-10-07 10:10:31 UTC
Github openshift ovirt-csi-driver-operator pull 26 0 None closed Bug 1868339: Don't create CredentialsRequest in ovirt-csi-driver-operator 2020-10-07 10:10:30 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:28:21 UTC

Description Yunfei Jiang 2020-08-12 10:49:33 UTC
According to the document https://github.com/openshift/cloud-credential-operator/blob/master/docs/mode-manual-creds.md

all required CredentialRequest for config CCO in manual mode should be extracted by command `oc adm release extract`.

there is a CredentialRequest in openshift-cluster-csi-drivers, but it is missing in the above oc comnand, this will cause installation failed when config CCO in manual mode, since user do not know the Secret for openshift-cluster-csi-drivers should be provided.

After provide Secret for openshift-cluster-csi-drivers, the cluster could be installed successfully.

>> errors if do not provide Secret for openshift-cluster-csi-drivers:

level=info msg="Cluster operator storage Progressing is True with AWSEBSCSIDriverOperatorCR_AWSEBSDriverController_AsExpected: AWSEBSCSIDriverOperatorCRProgressing: AWSEBSDriverControllerProgressing: Waiting for Deployment to deploy controller pods"
level=info msg="Cluster operator storage Available is False with AWSEBSCSIDriverOperatorCR_AWSEBSDriverController_AsExpected: AWSEBSCSIDriverOperatorCRAvailable: AWSEBSDriverControllerAvailable: Waiting for Deployment to deploy the CSI Controller Service"
level=fatal msg="failed to initialize the cluster: Cluster operator storage has not yet reported success"


>> CredentialRequest for openshift-cluster-csi-drivers:
---
oc describe CredentialsRequest -n openshift-cloud-credential-operator openshift-cluster-csi-drivers
Name:         openshift-cluster-csi-drivers
Namespace:    openshift-cloud-credential-operator
Labels:       <none>
Annotations:  operator.openshift.io/spec-hash: ccbc9a71628daecd6066ff25e09d562be0a5f649dbbe44fb3c10ba39fd3616cf
API Version:  cloudcredential.openshift.io/v1
Kind:         CredentialsRequest
Metadata:
  Creation Timestamp:  2020-08-12T02:50:07Z
  Generation:          1
  Managed Fields:
    API Version:  cloudcredential.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:operator.openshift.io/spec-hash:
      f:spec:
        .:
        f:providerSpec:
          .:
          f:apiVersion:
          f:kind:
          f:statementEntries:
        f:secretRef:
          .:
          f:name:
          f:namespace:
    Manager:         aws-ebs-csi-driver-operator
    Operation:       Update
    Time:            2020-08-12T02:50:07Z
  Resource Version:  28521
  Self Link:         /apis/cloudcredential.openshift.io/v1/namespaces/openshift-cloud-credential-operator/credentialsrequests/openshift-cluster-csi-drivers
  UID:               f58b58bd-896f-4449-8242-b605c61ef512
Spec:
  Provider Spec:
    API Version:  cloudcredential.openshift.io/v1
    Kind:         AWSProviderSpec
    Statement Entries:
      Action:
        ec2:AttachVolume
        ec2:CreateSnapshot
        ec2:CreateTags
        ec2:CreateVolume
        ec2:DeleteSnapshot
        ec2:DeleteTags
        ec2:DeleteVolume
        ec2:DescribeInstances
        ec2:DescribeSnapshots
        ec2:DescribeTags
        ec2:DescribeVolumes
        ec2:DescribeVolumesModifications
        ec2:DetachVolume
        ec2:ModifyVolume
      Effect:    Allow
      Resource:  *
  Secret Ref:
    Name:       aws-cloud-credentials
    Namespace:  openshift-cluster-csi-drivers
Events:         <none>


Version-Release number of the following components: 
4.6.0-0.nightly-2020-08-10-180431
 
How reproducible: 
Always 
 
Steps to Reproduce: 
1. Create install-config
2. Config CCO in manual mode, refer to https://github.com/openshift/cloud-credential-operator/blob/master/docs/mode-manual-creds.md
2. Create cluster
 
Actual results: 
create cluster failed:

level=info msg="Cluster operator storage Progressing is True with AWSEBSCSIDriverOperatorCR_AWSEBSDriverController_AsExpected: AWSEBSCSIDriverOperatorCRProgressing: AWSEBSDriverControllerProgressing: Waiting for Deployment to deploy controller pods"
level=info msg="Cluster operator storage Available is False with AWSEBSCSIDriverOperatorCR_AWSEBSDriverController_AsExpected: AWSEBSCSIDriverOperatorCRAvailable: AWSEBSDriverControllerAvailable: Waiting for Deployment to deploy the CSI Controller Service"
level=fatal msg="failed to initialize the cluster: Cluster operator storage has not yet reported success"

Expected results: 
create cluster succesfully

Additional info:

Comment 1 Jan Chaloupka 2020-08-19 16:21:44 UTC
Hi Yunfei Jiang,

I am not able to currently pull a release image. Figuring out where to get the permissions.

Meantime, can you more elaborate on what's happening?

> there is a CredentialRequest in openshift-cluster-csi-drivers, but it is missing in the above oc comnand, this will cause installation failed when config CCO in manual mode, since user do not know the Secret for openshift-cluster-csi-drivers should be provided.

Are you saying that after running `oc adm release extract quay.io/openshift-release-dev/ocp-release:4.4.6-x86_64 --to ./release-image` (as described in https://github.com/openshift/cloud-credential-operator/blob/master/docs/mode-manual-creds.md), openshift-cluster-csi-drivers/openshift-cloud-credential-operator CredentialsRequest is completely missing from the extracted manifests?

Comment 4 Hemant Kumar 2020-08-25 14:35:21 UTC
*** Bug 1871713 has been marked as a duplicate of this bug. ***

Comment 5 Fabio Bertinatto 2020-08-28 15:42:53 UTC
A PR has been created to add the CredentialsRequests to the payload: https://github.com/openshift/cluster-storage-operator/pull/80

Once this is merged, subsequent PRs to CSI driver operator repos to NOT create those objects themselves are needed before we can verify this bug:

AWS EBS: https://github.com/openshift/aws-ebs-csi-driver-operator/blob/master/pkg/operator/starter.go#L70-L75
oVirt: https://github.com/openshift/ovirt-csi-driver-operator/blob/master/pkg/operator/starter.go#L86-L91
Manila: https://github.com/openshift/csi-driver-manila-operator/blob/master/pkg/operator/starter.go#L91-L96

Comment 8 Fabio Bertinatto 2020-09-01 08:29:20 UTC
Moving this back to ASSIGNED because other PRs to CSI driver operator repos are necessary to complete the transition.

Comment 9 Fabio Bertinatto 2020-09-01 10:46:42 UTC
Once all attached PRs are merged:

1) All CredentialsRequest should be applied by CVO
2) All CredentialsRequest should be available in the image payload
2) No CredentialsRequest will be applied by CSI driver operators (EBS, Manila and oVirt)

As part of the verification of this PR, please confirm if all of those CSI driver operators are installed and correctly.

Comment 11 Yunfei Jiang 2020-09-03 08:59:56 UTC
PRs LGTM, new CR was added with correct permission:

version: 4.6.0-0.nightly-2020-09-03-005025

>> oc adm release extract $buildimage --cloud aws --credentials-requests

<--snip-->
---
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: aws-ebs-csi-driver-operator
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - action:
      - ec2:AttachVolume
      - ec2:CreateSnapshot
      - ec2:CreateTags
      - ec2:CreateVolume
      - ec2:DeleteSnapshot
      - ec2:DeleteTags
      - ec2:DeleteVolume
      - ec2:DescribeInstances
      - ec2:DescribeSnapshots
      - ec2:DescribeTags
      - ec2:DescribeVolumes
      - ec2:DescribeVolumesModifications
      - ec2:DetachVolume
      - ec2:ModifyVolume
      effect: Allow
      resource: '*'
  secretRef:
    name: ebs-cloud-credentials
    namespace: openshift-cluster-csi-drivers
<--snip-->

>> oc adm release extract registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-03-005025 --to ./release-image

0000_50_cluster-storage-operator_03_credentials_request_aws.yaml was extracted.

>> grep -l "apiVersion: cloudcredential.openshift.io" * | xargs cat

<--snip-->
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: aws-ebs-csi-driver-operator
  namespace: openshift-cloud-credential-operator
spec:
  secretRef:
    name: ebs-cloud-credentials
    namespace: openshift-cluster-csi-drivers
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - effect: Allow
      action:
      - ec2:AttachVolume
      - ec2:CreateSnapshot
      - ec2:CreateTags
      - ec2:CreateVolume
      - ec2:DeleteSnapshot
      - ec2:DeleteTags
      - ec2:DeleteVolume
      - ec2:DescribeInstances
      - ec2:DescribeSnapshots
      - ec2:DescribeTags
      - ec2:DescribeVolumes
      - ec2:DescribeVolumesModifications
      - ec2:DetachVolume
      - ec2:ModifyVolume
      #- ec2:*
      resource: "*"
<--snip-->

The 4.5 -> 4.6 upgrade process when CCO in manual mode is tracked by https://bugzilla.redhat.com/show_bug.cgi?id=1871713#c7

Comment 12 Qin Ping 2020-09-03 14:36:06 UTC
Hello Fabio,

These PRs make aws ebs csi driver operator and oVirt csi driver operator working well, but introduced a bug for manila csi driver operator.

The detail is:
1. The credentailsrequest created by manila csi driver operator,  its secret `manila-cloud-credentials` will be created in `openshift-manila-csi-driver` namespace.(https://github.com/openshift/csi-driver-manila-operator/pull/57/files line#8-9)
2. But the credentailsrequest created by CVO, its secret `manila-cloud-credentials` wil be created in `openshift-cluster-csi-drivers` namespace.(https://github.com/bertinatto/cluster-storage-operator/blob/2c521c2d7f18a804c1fc7360a9778dc3bc9dea99/manifests/03_credentials_request_openstack.yaml#L9)

So, PV dynamic provisioning does not work now.
 Warning  ProvisioningFailed    2m28s (x12 over 20m)  manila.csi.openstack.org_openstack-manila-csi-controllerplugin-6c65fb77f7-djspk_a8b9526c-e510-4771-8338-fe4b64ecd998  failed to provision volume with StorageClass "csi-manila-ceph": error getting secret csi-manila-secrets in namespace openshift-manila-csi-driver: secrets "csi-manila-secrets" not found

Check the opetor logs, it always pending to "Waiting for secret manila-cloud-credentials from cloud-credentials-operator"

So, I'll reassign this bug back.

Comment 15 Qin Ping 2020-09-04 01:02:09 UTC
verified with: 4.6.0-0.nightly-2020-09-03-224310

Comment 17 errata-xmlrpc 2020-10-27 16:28:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.