Bug 2068433
| Summary: | Can't upload files when configure s3 storage for internal registry with registry credentials on vsphere | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Container Storage | Reporter: | XiuJuan Wang <xiuwang> |
| Component: | Multi-Cloud Object Gateway | Assignee: | Nimrod Becker <nbecker> |
| Status: | CLOSED NOTABUG | QA Contact: | Ben Eli <belimele> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | unspecified | CC: | aos-bugs, etamir, jsafrane, jthottan, ocs-bugs |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-04-08 09:46:20 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1976782, 2060362 | ||
Let me cut down the reproduce steps, the registry configure and builds are not necessary conditions.
Steps to Reproduce:
1.Installed a vsphere cluster with 3 workers, each worker has 10cpu and 24G memory
Install odf operator, and install StorageSystem
Expose the ceph RGW service.
$oc expose svc rook-ceph-rgw-ocs-storagecluster-cephobjectstore --hostname=rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.wxjbigvsb.qe.devcluster.openshift.com
Create an obc named "my314" using the ceph RGW
$ AWS_ACCESS_KEY_ID=$(oc get secret my314 -n openshift-storage -o yaml | grep -w "AccessKey:" | head -n1 | awk '{print $2}' | base64 --decode)
$ AWS_SECRET_ACCESS_KEY=$(oc get secret my314 -n openshift-storage -o yaml | grep -w "SecretKey:" | head -n1 | awk '{print $2}' | base64 --decode)
2. Check the aws s3 api
$aws s3 --no-verify-ssl --endpoint http://rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.wxjbigvsb.qe.devcluster.openshift.com cp ./bb.json s3://my314-271e8d37-08f9-4d19-a8e7-7baf734428a2/bb.json
upload failed: ./bb.json to s3://my314-271e8d37-08f9-4d19-a8e7-7baf734428a2/bb.json An error occurred (AccessDenied) when calling the PutObject operation: Unknown
Moving to Noobaa team. |
Description of problem: Can't upload files when configure s3 storage for internal registry with registry credentials on vsphere, it prompts 403 error. Version-Release number of selected component (if applicable): 4.11.0 How reproducible: always Steps to Reproduce: 1.Installed a vsphere cluster with 3 workers, each worker has 10cpu and 24G memory Install odf operator, and install StorageSystem Expose the ceph RGW service. $oc expose svc rook-ceph-rgw-ocs-storagecluster-cephobjectstore --hostname=rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.wxjbigvsb.qe.devcluster.openshift.com Create an obc "my314" using the ceph RGW $ AWS_ACCESS_KEY_ID=$(oc get secret my314 -n openshift-storage -o yaml | grep -w "AccessKey:" | head -n1 | awk '{print $2}' | base64 --decode) $ AWS_SECRET_ACCESS_KEY=$(oc get secret my314 -n openshift-storage -o yaml | grep -w "SecretKey:" | head -n1 | awk '{print $2}' | base64 --decode) 2. Configure registrty credentials $oc create secret generic image-registry-private-configuration-user --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=${AWS_ACCESS_KEY_ID} --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=${AWS_SECRET_ACCESS_KEY} --namespace openshift-image-registry $oc patch config.image/cluster -p '{"spec":{"storage":{"S3":{"bucket":"my314-271e8d37-08f9-4d19-a8e7-7baf734428a2","encrypt":true,"region":"us-east-1","regionEndpoint":"http://rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.wxjbigvsb.qe.devcluster.openshift.com","virtualHostedStyle":false}}}}' --type=merge sleep 60 $oc get config.image/cluster -o jsonpath='{.spec.storage.s3.bucket}' my314-271e8d37-08f9-4d19-a8e7-7baf734428a2 3. Check the aws s3 api $aws s3 --no-verify-ssl --endpoint http://rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.wxjbigvsb.qe.devcluster.openshift.com ls s3://my314-271e8d37-08f9-4d19-a8e7-7baf734428a2 $aws s3 --profile=test --no-verify-ssl --endpoint http://rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.wxjbigvsb.qe.devcluster.openshift.com cp ./bb.json s3://my314-271e8d37-08f9-4d19-a8e7-7baf734428a2/bb.json upload failed: ./bb.json to s3://my314-271e8d37-08f9-4d19-a8e7-7baf734428a2/bb.json An error occurred (AccessDenied) when calling the PutObject operation: Unknown $oc get builds -n wxj1 httpd-ex-7 Source Git@753f06d Failed (PushImageToRegistryFailed) 11 minutes ago 2m29s oc get config.image -o yaml apiVersion: v1 items: - apiVersion: imageregistry.operator.openshift.io/v1 kind: Config metadata: creationTimestamp: "2022-03-14T08:04:56Z" finalizers: - imageregistry.operator.openshift.io/finalizer generation: 14 name: cluster resourceVersion: "292633" uid: 2b6c5bcd-63fa-4894-bb8d-b3fe8d258e2d spec: httpSecret: b713a4ae5bd568dd8266f944b1d109768c08105711e80e7b2a8f9e5da58be8c8d70f44f7cf96c17bb3c4c89472111600365525f46c1da08c421e5acf70879ee8 logLevel: Normal managementState: Managed observedConfig: null operatorLogLevel: Normal proxy: {} readOnly: false replicas: 2 requests: read: maxWaitInQueue: 0s write: maxWaitInQueue: 0s rolloutStrategy: RollingUpdate storage: managementState: Unmanaged s3: bucket: my314-271e8d37-08f9-4d19-a8e7-7baf734428a2 encrypt: true region: us-east-1 regionEndpoint: http://rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.wxjbigvsb.qe.devcluster.openshift.com virtualHostedStyle: false unsupportedConfigOverrides: null status: conditions: - lastTransitionTime: "2022-03-14T10:38:04Z" message: The registry is ready reason: Ready status: "False" type: Progressing - lastTransitionTime: "2022-03-14T10:37:37Z" message: The registry is ready reason: Ready status: "True" type: Available - lastTransitionTime: "2022-03-14T08:04:56Z" status: "False" type: Degraded - lastTransitionTime: "2022-03-14T08:05:54Z" status: "False" type: Removed - lastTransitionTime: "2022-03-14T08:04:58Z" reason: AsExpected status: "False" type: ImageRegistryCertificatesControllerDegraded - lastTransitionTime: "2022-03-14T08:04:58Z" message: The daemon set node-ca has available replicas reason: AsExpected status: "True" type: NodeCADaemonAvailable - lastTransitionTime: "2022-03-14T08:04:58Z" reason: AsExpected status: "False" type: NodeCADaemonControllerDegraded - lastTransitionTime: "2022-03-14T08:04:59Z" reason: AsExpected status: "False" type: ImageConfigControllerDegraded - lastTransitionTime: "2022-03-14T10:20:18Z" reason: S3 Bucket Exists status: "True" type: StorageExists - lastTransitionTime: "2022-03-14T08:12:32Z" reason: AsExpected status: "False" type: AzureStackCloudControllerDegraded generations: - group: apps hash: "" lastGeneration: 0 name: node-ca namespace: openshift-image-registry resource: daemonsets - group: apps hash: "" lastGeneration: 12 name: image-registry namespace: openshift-image-registry resource: deployments observedGeneration: 14 readyReplicas: 2 storage: managementState: Unmanaged s3: bucket: my314-271e8d37-08f9-4d19-a8e7-7baf734428a2 encrypt: true region: us-east-1 regionEndpoint: http://rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.wxjbigvsb.qe.devcluster.openshift.com virtualHostedStyle: false storageManaged: false kind: List metadata: resourceVersion: "" selfLink: "" Registry log http://virt-openshift-05.lab.eng.nay.redhat.com/xiuwang/bz1976782/registry.log Expected results: Could upload files to s3 storage with registry credentials. Additional info: