Bug 1402443

Is this currently a supported option for 3.2 (using S3 as the storage for the internal registry, and using the AWS IAM system for authentication)?

A colleague has tried a similar setup on OSCP 3.3 without any issues.

Error since on the original system

time="2016-12-02T10:10:35.356540692-05:00" level=error msg="response completed with error" err.code=UNKNOWN err.detail="s3: The AWS Access Key Id you provided does not exist in our records." err.message="unknown error" go.version=go1.4.2 http.request.host="172.30.132.220:5000" http.request.id=8d4ff520-8bf3-4dcc-aa33-a127dc6206f2 http.request.method=HEAD http.request.remoteaddr="10.1.2.1:55494" http.request.uri="/v2/openshift/registry-console/blobs/sha256:9b0dee6356a152d6cbac4adf6ce40ae4df40442d559931862ad65c78ed373979" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/79ebcd8-unsupported kernel/3.10.0-327.36.3.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=10.296812899s http.response.status=500 http.response.written=358 instance.id=0cf6668f-11c0-44a9-b146-3d48b477cf62 vars.digest="sha256:9b0dee6356a152d6cbac4adf6ce40ae4df40442d559931862ad65c78ed373979" vars.name="openshift/registry-console"
 
Running OCP 3.2.1.17 on RHEL 7.2

Hello Anthony, thank you for the input. Is there any manual or article about how to configure it with version 3.3 ? Especially, how to configure the IAM configuration.

Thank you

Hi,

Couldn't find a good article/guide to setting this up, but I did get it running ok with an OpenShift 3.3 installation, using the config shown below.

Would be good to clarify if this not expected to work with OpenShift versions prior to 3.3.


Registry Config — the following YAML is mounted in the registry pod as a secret:

version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  s3:
    accesskey: 
    secretkey: 
    region: eu-west-1 
    bucket: friday-oscp-registry-test 
    encrypt: true
    keyid: 21ce6234-9da6-491d-83e5-ae06e1ff72cb
    secure: true
    v4auth: true 
    rootdirectory: /registry
    chunksize: "26214400"
auth:
  openshift:
    realm: openshift
middleware:
  registry:
  - name: openshift
  repository:
  - name: openshift
    options:
      pullthrough: True
      acceptschema2: True
      enforcequota: True
  storage:
  - name: openshift



IAM/KMS Config:


Policy:  test-s3-openshift

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::friday-oscp-registry-test"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
            ],
            "Resource": "arn:aws:s3:::friday-oscp-registry-test/*"
        }
    ]
}

This policy has the following role attached (you can also see this as the policy having the role attached).

Role:  dev-NodePolicy-F16K2HHVJRRF

This role is the one which was associated with the EC2 instances at creation of the node instances.  The role can be updated after launch, but you cannot create a new role and associate it with an already running EC2 instance.


S3 Bucket:  friday-oscp-registry-test

Has this policy in the properties->permissions tab in S3:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::721160691850:role/dev-NodePolicy-F16K2HHVJRRF"
			},
			"Action": [
				"s3:ListBucket",
				"s3:GetBucketLocation",
				"s3:ListBucketMultipartUploads"
			],
			"Resource": "arn:aws:s3:::friday-oscp-registry-test"
		},
		{
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::721160691850:role/dev-NodePolicy-F16K2HHVJRRF"
			},
			"Action": [
				"s3:PutObject",
				"s3:GetObject",
				"s3:DeleteObject",
				"s3:ListMultipartUploadParts",
				"s3:AbortMultipartUpload"
			],
			"Resource": "arn:aws:s3:::friday-oscp-registry-test/*"
		}
	]
}



Encryption Key — in the EU-West-1 region

Key name:  oscp-registry-key 
ARN:  arn:aws:kms:eu-west-1:721160691850:key/21ce6234-9da6-491d-83e5-ae06e1ff72cb 

The key has the dev-NodePolicy-F16K2HHVJRRF role attached (as both admin and as an encryption user, though admin should probably be a different role).

I've done some more testing.

Using the above config for KMS/IAM with an OSCP 3.2 installation fails, with the same error the customer is seeing:

[root@ip-172-31-15-84 ec2-user]# oc version
oc v3.2.1.21
kubernetes v1.2.0-36-g4a3f9c5


time="2016-12-14T11:17:52.109785672Z" level=error msg="response completed with error" err.code=UNKNOWN err.detail="s3: The AWS Access Key Id you provided does not exist in our records." err.message="unknown error" go.version=go1.4.2 http.request.host=" 172.30.206.90:5000 " http.request.id =dbe46f6b-82c2-4d64-a196-95c1915a82d2 http.request.method=HEAD http.request.remoteaddr=" 10.1.0.1:35140 " http.request.uri="/v2/test/ruby-hello-world/blobs/sha256:08d48e6f1cff259389732d35307bb877215fa28867cdaff50c1dbd6e0b993c1f" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/429be27-unsupported kernel/3.10.0-327.36.3.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=27.061049ms http.response.status=500 http.response.written=358 instance.id =7ac22d86-5e77-4eb8-a138-45db1df3a319 vars.digest="sha256:08d48e6f1cff259389732d35307bb877215fa28867cdaff50c1dbd6e0b993c1f" vars.name ="test/ruby-hello-world"
10.1.0.1 - - [14/Dec/2016:11:17:52 +0000] "HEAD /v2/test/ruby-hello-world/blobs/sha256:08d48e6f1cff259389732d35307bb877215fa28867cdaff50c1dbd6e0b993c1f HTTP/1.1" 500 358 "" "docker/1.10.3 go/go1.6.2 git-commit/429be27-unsupported kernel/3.10.0-327.36.3.el7.x86_64 os/linux arch/amd64"



However, if I pull the latest docker-registry image and use that, then the S3 authentication works fine and I can push into the registry.

Version of the image is:

"Name": "openshift3/ose-docker-registry",
"Release": "0",
"Vendor": "Red Hat, Inc.",
"Version": "v3.3.1.7",
"architecture": "x86_64",
"authoritative-source-url": " registry.access.redhat.com ",
"build-date": "2016-12-01T20:05:17.228640",



So this would suggest that the fix to allow the keyid to work with a role was introduced to the docker-registry image for 3.3.

I've checked the changelog for package atomic-openshift-dockerregistry (which supplies the registry binaries) but I cannot see a direct reference to the S3 fix in there.  Would be good to confirm when/how this fix went in.

Thanks,
Tony

Not sure what fix went in, but if this is working let's verify it and close it.

Test with
openshift v3.7.0-0.191.0
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

cannot push image to registry

# oc logs -f po/docker-registry-4-7scx7

time="2017-11-07T09:28:19.71473391Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: NoCredentialProviders: no valid providers in chain. Deprecated. \n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" err.message="unknown error" go.version=go1.8.3 http.request.host="docker-registry.default.svc:5000" http.request.id=2764c9f3-1695-4187-8470-8e958c308e48 http.request.method=POST http.request.remoteaddr="10.2.2.1:44830" http.request.uri="/v2/install-test/nodejs-mongodb-example/blobs/uploads/" http.request.useragent="docker/1.12.6 go/go1.8.3 kernel/3.10.0-693.el7.x86_64 os/linux arch/amd64 UpstreamClient(go-dockerclient)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=28.137132ms http.response.status=500 http.response.written=104 instance.id=2dcd8a5d-bca2-448c-a352-44df4035af15 openshift.auth.user="system:serviceaccount:install-test:builder" openshift.logger=registry vars.name="install-test/nodejs-mongodb-example"

upstream issue to support this in the driver: https://github.com/docker/distribution/issues/1684