After configure registry using ceph rgw , met 403 error, blocked by https://bugzilla.redhat.com/show_bug.cgi?id=2068433 time="2022-04-01T06:16:49.900908996Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: AccessDenied: \n\tstatus code: 403, request id: tx0000000000000000002c0-00624698d1-89fc-ocs-storagecluster-cephobjectstore, host id: " err.message="unknown error" go.version=go1.17.5 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=99d5be30-4de1-442e-b785-554f81066883 http.request.method=POST http.request.remoteaddr="10.129.2.16:40036" http.request.uri=/v2/default/httpd-ex/blobs/uploads/ http.request.useragent="containers/5.19.1 (github.com/containers/image)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=50.836091ms http.response.status=500 http.response.written=123 openshift.auth.user="system:serviceaccount:default:builder" vars.name=default/httpd-ex
Steps to Reproduce: 1.Installed a vsphere cluster with 3 workers, each worker has 10cpu and 24G memory Install odf operator, and install StorageSystem Create an obc named "jitli-object-bucket" using the ceph RGW get the Object Bucket Claim Data Expose the ceph RGW service. redhat@jitli:~$ oc expose svc rook-ceph-rgw-ocs-storagecluster-cephobjectstore --hostname=rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.jitlivs411a.qe.devcluster.openshift.com -n openshift-storage route.route.openshift.io/rook-ceph-rgw-ocs-storagecluster-cephobjectstore exposed get the Object Bucket Claim Data and create secret redhat@jitli:~$ export AWS_ACCESS_KEY_ID= redhat@jitli:~$ export AWS_SECRET_ACCESS_KEY= redhat@jitli:~$ oc create secret generic image-registry-private-configuration-user --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=${AWS_ACCESS_KEY_ID} --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=${AWS_SECRET_ACCESS_KEY} --namespace openshift-image-registry secret/image-registry-private-configuration-user created 2. Check the aws s3 api redhat@jitli:~$ aws s3 --no-verify-ssl --endpoint http://rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.jitlivs411a.qe.devcluster.openshift.com ls 2022-04-11 13:32:57 jitli-object-bucket-5ec98c5e-7b17-4045-9e3f-88109de770ff redhat@jitli:~$ aws s3 --no-verify-ssl --endpoint http://rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.jitlivs411a.qe.devcluster.openshift.com cp ./bb.json s3://jitli-object-bucket-5ec98c5e-7b17-4045-9e3f-88109de770ff/bb.json upload: ./bb.json to s3://jitli-object-bucket-5ec98c5e-7b17-4045-9e3f-88109de770ff/bb.json redhat@jitli:~$ oc edit config.image spec: ... storage: managementState: Unmanaged s3: s3: bucket: jitli-object-bucket-5ec98c5e-7b17-4045-9e3f-88109de770ff region: us-east-1 regionEndpoint: http://rook-ceph-rgw-ocs-storagecluster-openshift-storage.apps.jitlivs411a.qe.devcluster.openshift.com virtualHostedStyle: false ... redhat@jitli:~$ oc new-app httpd~http://github.com/openshift/httpd-ex.git -n default --name='jitli' redhat@jitli:~$ oc get builds -n default NAME TYPE FROM STATUS STARTED DURATION jitli-1 Source Git@753f06d Complete 58 seconds ago 42s redhat@jitli:~$ oc logs -f build/jitli-1 -n default ... Successfully pushed image-registry.openshift-image-registry.svc:5000/default/jitli@sha256:623c0f14b5439f9d5afdfbe1d92076c93a787bc8f4bbfade73b0c447c287651e Push successful
Verified on 4.10.0-0.nightly-2022-04-07-042325 cluster, see details steps in comment #5
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.10 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:1356