Description of problem: After create a imagestreamimport with a private docker image, the master log display: {imagestreamimports /v1, Kind=ImageStreamImport 0x45b0840 0xc2081b0880 0xc20810a650 {}} I0128 00:41:38.507992 20064 importer.go:684] Falling back to an HTTP check for an insecure registry {https <nil> test.com:80 }: Get https://test.com:80/v2/: tls: o versized record received with length 20527 I0128 00:41:38.514300 20064 credentials.go:108] Unable to find a secret to match http://test.com:80/v2/test/busybox/tags/list (test.com:80/v2/test/busybox/tags/list) I0128 00:41:38.514324 20064 importer.go:348] unable to access tags for repository &importer.importRepository{Ref:api.DockerImageReference{Registry:"test.com:80", Name space:"test", Name:"busybox", Tag:"", ID:""}, Registry:(*url.URL)(0xc20f0dcb80), Name:"test/busybox", Insecure:true, Tags:[]importer.importTag(nil), Digests:[]importer. importDigest(nil), MaximumTags:5, AdditionalTags:[]string(nil), Err:error(nil)}: &url.Error{Op:"Get", URL:"http://test.com:80/v2/test/busybox/tags/list", Err:(*errors.e rrorString)(0xc20dece750)} I0128 00:41:38.526301 20064 image_change_controller.go:47] Build image change controller detected ImageStream change -------------------------- And when check all the imagestream secrets of the imagestream: # curl -k --cert /openshift.local.config/master/admin.crt --key /openshift.local.config/master/admin.key https://127.0.0.1:8443/oapi/v1/namespaces/wjiang/imagestreams/bc/secrets ...... { "metadata": { "name": "test", "namespace": "wjiang", "selfLink": "/api/v1/namespaces/wjiang/secrets/test", "uid": "5a1cb645-c56a-11e5-8b5f-0ef77b0359c5", "resourceVersion": "583", "creationTimestamp": "2016-01-28T02:53:47Z" }, "data": { ".dockercfg": "eyJ0ZXN0LmNvbTo4MCI6eyJ1c2VybmFtZSI6InRlc3QiLCJwYXNzd29yZCI6InRlc3QiLCJlbWFpbCI6InRlc3RAcmVkaGF0LmNvbSIsImF1dGgiOiJkR1Z6ZERwMFpYTjAifX0=" }, "type": "kubernetes.io/dockercfg" } ] } # echo "eyJ0ZXN0LmNvbTo4MCI6eyJ1c2VybmFtZSI6InRlc3QiLCJwYXNzd29yZCI6InRlc3QiLCJlbWFpbCI6InRlc3RAcmVkaGF0LmNvbSIsImF1dGgiOiJkR1Z6ZERwMFpYTjAifX0="|base64 -d {"test.com:80":{"username":"test","password":"test","email":"test","auth":"dGVzdDp0ZXN0"}} Version-Release number of selected component (if applicable): devenv-rhel7_3259 How reproducible: always Steps to Reproduce: 1. Setup a private docker registry 1> docker run -d registry:2 2> docker run -d -v /nginx.conf:/etc/nginx/conf.d/registry.conf -v /htpasswd:/etc/nginx/.htpasswd nginx # cat nginx.conf server { listen 80; server_name test.com; add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always; location / { proxy_pass http://172.17.0.1:5000; auth_basic "Restricted"; auth_basic_user_file /etc/nginx/.htpasswd; } } 3> Add the insecure registry to /etc/sysconfig/docker & restart docker service OPTIONS='--insecure-registry 172.30.0.0/16 --insecure-registry 172.17.0.0/16 --selinux-enabled' 2.Create a imagestreamimport # oc create -f isi { "kind": "ImageStreamImport", "apiVersion": "v1", "metadata": { "name": "bc" }, "spec":{ "import": true, "repository": { "from": {"kind": "DockerImage", "name": "test.com:80/test/busybox"}, "importPolicy": {"insecure": true } }} } 3. Check the imagestream & check the master log (loglevel=5) Actual results: 3. 1> # oc describe is bc Name: bc Created: 34 minutes ago Labels: <none> Annotations: openshift.io/image.dockerRepositoryCheck=2016-01-28T05:52:13Z Docker Pull Spec: <none> Tags: <none> 2> {imagestreamimports /v1, Kind=ImageStreamImport 0x45b0840 0xc2081b0880 0xc20810a650 {}} I0128 00:41:38.507992 20064 importer.go:684] Falling back to an HTTP check for an insecure registry {https <nil> test.com:80 }: Get https://test.com:80/v2/: tls: o versized record received with length 20527 I0128 00:41:38.514300 20064 credentials.go:108] Unable to find a secret to match http://test.com:80/v2/test/busybox/tags/list (test.com:80/v2/test/busybox/tags/list) I0128 00:41:38.514324 20064 importer.go:348] unable to access tags for repository &importer.importRepository{Ref:api.DockerImageReference{Registry:"test.com:80", Name space:"test", Name:"busybox", Tag:"", ID:""}, Registry:(*url.URL)(0xc20f0dcb80), Name:"test/busybox", Insecure:true, Tags:[]importer.importTag(nil), Digests:[]importer. importDigest(nil), MaximumTags:5, AdditionalTags:[]string(nil), Err:error(nil)}: &url.Error{Op:"Get", URL:"http://test.com:80/v2/test/busybox/tags/list", Err:(*errors.e rrorString)(0xc20dece750)} I0128 00:41:38.526301 20064 image_change_controller.go:47] Build image change controller detected ImageStream change Expected results: Should work well Additional info:
Here is the result when access the registry api: # curl http://test:test@test.com/v2/test/busybox/tags/list {"name":"test/busybox","tags":["latest"]}
When I try to import imagestream which source is a https scheme private registry, also failed even I add insecure=true { ImageStreamImport} &{imagestreamimports /v1, Kind=ImageStreamImport 0x45aa7c0 0xc2081b08e0 0xc20810a650 {}} I0129 03:35:01.605214 4013 importer.go:684] Falling back to an HTTP check for an insecure registry {https <nil> test.com:5000 }: Get https://test.com:5000/v2/: x5 09: certificate signed by unknown authority I0129 03:35:01.608390 4013 importer.go:310] unable to access repository &importer.importRepository{Ref:api.DockerImageReference{Registry:"test.com:5000", Namespace:" test", Name:"busybox", Tag:"", ID:""}, Registry:(*url.URL)(0xc20a634780), Name:"test/busybox", Insecure:true, Tags:[]importer.importTag(nil), Digests:[]importer.importD igest(nil), MaximumTags:5, AdditionalTags:[]string(nil), Err:error(nil)}: &url.Error{Op:"Get", URL:"https://test.com:5000/v2/", Err:x509.UnknownAuthorityError{cert:(*x5 09.Certificate)(0xc20b54d680), hintErr:error(nil), hintCert:(*x509.Certificate)(nil)}} I0129 03:35:01.617447 4013 image_change_controller.go:47] Build image change controller detected ImageStream change 1. setup https docker registry 1> docker run --entrypoint htpasswd registry:2 -Bbn test test > htpasswd 2> oadm ca create-server-cert --cert=server.crt --key=server.key --hostnames=test.com 3> add --insecure-registry test.com:5000 to /etc/sysconfig/docker 4> docker run -d --restart=always --name registry -v /htpasswd:/htpasswd -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/htpasswd -v /server.crt:/server.crt -v /server.key:/server.key -e REGISTRY_HTTP_TLS_CERTIFICATE=/server.crt -e REGISTRY_HTTP_TLS_KEY=/server.key registry:2 2. prepare testing data 1> docker pull busybox 2> docker tag busybox:latest test.com:5000/test/busybox:latest 3> docker push test.com:5000/test/busybox (with user credentials) 4> # curl -k https://test:test@test.com:5000/v2/_catalog {"repositories":["test/busybox"]} 5> # curl -k https://test:test@test.com:5000/v2/test/busybox/tags/list {"name":"test/busybox","tags":["latest"]} 3. create imagestreamimport { "kind": "ImageStreamImport", "apiVersion": "v1", "metadata": { "name": "tete" }, "spec":{ "import": true, "repository": { "from": {"kind": "DockerImage", "name": "test.com:5000/test/busybox"}, "importPolicy": {"insecure": true } }} } 4. check the master log (loglevel=5) { ImageStreamImport} &{imagestreamimports /v1, Kind=ImageStreamImport 0x45aa7c0 0xc2081b08e0 0xc20810a650 {}} I0129 03:35:01.605214 4013 importer.go:684] Falling back to an HTTP check for an insecure registry {https <nil> test.com:5000 }: Get https://test.com:5000/v2/: x5 09: certificate signed by unknown authority I0129 03:35:01.608390 4013 importer.go:310] unable to access repository &importer.importRepository{Ref:api.DockerImageReference{Registry:"test.com:5000", Namespace:" test", Name:"busybox", Tag:"", ID:""}, Registry:(*url.URL)(0xc20a634780), Name:"test/busybox", Insecure:true, Tags:[]importer.importTag(nil), Digests:[]importer.importD igest(nil), MaximumTags:5, AdditionalTags:[]string(nil), Err:error(nil)}: &url.Error{Op:"Get", URL:"https://test.com:5000/v2/", Err:x509.UnknownAuthorityError{cert:(*x5 09.Certificate)(0xc20b54d680), hintErr:error(nil), hintCert:(*x509.Certificate)(nil)}} I0129 03:35:01.617447 4013 image_change_controller.go:47] Build image change controller detected ImageStream change
We weren't allowing insecure TLS connections, only HTTP connections. Fixed in https://github.com/openshift/origin/pull/6800
You may want to test this branch against your repo - it's surprisingly difficult to get good testing of insecure registries in a dev environment and it's possible the transport is still missing something.
(In reply to Clayton Coleman from comment #3) > We weren't allowing insecure TLS connections, only HTTP connections. Fixed > in https://github.com/openshift/origin/pull/6800 Checked on devenv-rhel7_3301, dockerhub work well.
Merged
Since devenv-rhel7_3320 is not QE_READY, and can not start openshift on it, so will delay checking this. #openshift start ...... E0203 04:20:53.382960 5986 ensure.go:201] error waiting for policy cache to initialize: timed out waiting for the condition E0203 04:20:53.383870 5986 ensure.go:164] Unable to create default security context constraint privileged. Got error: User "system:openshift-master" cannot create securitycontextconstraints at the cluster scope E0203 04:20:53.384535 5986 ensure.go:164] Unable to create default security context constraint nonroot. Got error: User "system:openshift-master" cannot create securitycontextconstraints at the cluster scope E0203 04:20:53.385204 5986 ensure.go:164] Unable to create default security context constraint hostmount-anyuid. Got error: User "system:openshift-master" cannot create securitycontextconstraints at the cluster scope E0203 04:20:53.385828 5986 ensure.go:164] Unable to create default security context constraint hostaccess. Got error: User "system:openshift-master" cannot create securitycontextconstraints at the cluster scope E0203 04:20:53.386443 5986 ensure.go:164] Unable to create default security context constraint restricted. Got error: User "system:openshift-master" cannot create securitycontextconstraints at the cluster scope E0203 04:20:53.387062 5986 ensure.go:164] Unable to create default security context constraint anyuid. Got error: User "system:openshift-master" cannot create securitycontextconstraints at the cluster scope E0203 04:20:53.387651 5986 ensure.go:105] Error adding service account roles to "default" namespace: User "system:openshift-master" cannot get namespaces in project "default" E0203 04:20:53.388307 5986 ensure.go:52] Error creating namespace openshift-infra: User "system:openshift-master" cannot create namespaces at the cluster scope 2016-02-03 04:20:53.389162 I | skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:53 [rcache 0] 2016-02-03 04:20:53.389177 I | skydns: ready for queries on cluster.local. for udp4://0.0.0.0:53 [rcache 0] I0203 04:20:53.489446 5986 run_components.go:176] DNS listening at 0.0.0.0:53 I0203 04:20:53.490365 5986 start_master.go:508] Controllers starting (*) I0203 04:20:53.633145 5986 start_node.go:180] Starting a node connected to https://172.18.3.132:8443 I0203 04:20:53.647910 5986 plugins.go:71] No cloud provider specified. I0203 04:20:53.647932 5986 start_node.go:257] Starting node ip-172-18-3-132.ec2.internal (v1.1.1-385-g2fa2261) F0203 04:20:53.656722 5986 run_components.go:405] Unable to initialize namespaces: unable to refresh the security allocation UID blocks: User "system:openshift-master" cannot list all namespaces in the cluster
(In reply to weiwei jiang from comment #0) > Description of problem: > After create a imagestreamimport with a private docker image, the master log > display: > {imagestreamimports /v1, Kind=ImageStreamImport 0x45b0840 0xc2081b0880 > 0xc20810a650 {}} > I0128 00:41:38.507992 20064 importer.go:684] Falling back to an HTTP check > for an insecure registry {https <nil> test.com:80 }: Get > https://test.com:80/v2/: tls: o > versized record received with length 20527 > I0128 00:41:38.514300 20064 credentials.go:108] Unable to find a secret to > match http://test.com:80/v2/test/busybox/tags/list > (test.com:80/v2/test/busybox/tags/list) > I0128 00:41:38.514324 20064 importer.go:348] unable to access tags for > repository > &importer.importRepository{Ref:api.DockerImageReference{Registry:"test.com: > 80", Name > space:"test", Name:"busybox", Tag:"", ID:""}, > Registry:(*url.URL)(0xc20f0dcb80), Name:"test/busybox", Insecure:true, > Tags:[]importer.importTag(nil), Digests:[]importer. > importDigest(nil), MaximumTags:5, AdditionalTags:[]string(nil), > Err:error(nil)}: &url.Error{Op:"Get", > URL:"http://test.com:80/v2/test/busybox/tags/list", Err:(*errors.e > rrorString)(0xc20dece750)} > I0128 00:41:38.526301 20064 image_change_controller.go:47] Build image > change controller detected ImageStream change > > -------------------------- > > And when check all the imagestream secrets of the imagestream: > # curl -k --cert /openshift.local.config/master/admin.crt --key > /openshift.local.config/master/admin.key > https://127.0.0.1:8443/oapi/v1/namespaces/wjiang/imagestreams/bc/secrets > ...... > { > "metadata": { > "name": "test", > "namespace": "wjiang", > "selfLink": "/api/v1/namespaces/wjiang/secrets/test", > "uid": "5a1cb645-c56a-11e5-8b5f-0ef77b0359c5", > "resourceVersion": "583", > "creationTimestamp": "2016-01-28T02:53:47Z" > }, > "data": { > ".dockercfg": > "eyJ0ZXN0LmNvbTo4MCI6eyJ1c2VybmFtZSI6InRlc3QiLCJwYXNzd29yZCI6InRlc3QiLCJlbWFp > bCI6InRlc3RAcmVkaGF0LmNvbSIsImF1dGgiOiJkR1Z6ZERwMFpYTjAifX0=" > }, > "type": "kubernetes.io/dockercfg" > } > ] > } > > > # echo > "eyJ0ZXN0LmNvbTo4MCI6eyJ1c2VybmFtZSI6InRlc3QiLCJwYXNzd29yZCI6InRlc3QiLCJlbWFp > bCI6InRlc3RAcmVkaGF0LmNvbSIsImF1dGgiOiJkR1Z6ZERwMFpYTjAifX0="|base64 -d > {"test.com:80":{"username":"test","password":"test","email":"test@redhat. > com","auth":"dGVzdDp0ZXN0"}} > > > > Version-Release number of selected component (if applicable): > devenv-rhel7_3259 > > How reproducible: > always > > Steps to Reproduce: > 1. Setup a private docker registry > 1> docker run -d registry:2 > 2> docker run -d -v /nginx.conf:/etc/nginx/conf.d/registry.conf -v > /htpasswd:/etc/nginx/.htpasswd nginx > # cat nginx.conf > server { > listen 80; > server_name test.com; > add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always; > location / { > proxy_pass http://172.17.0.1:5000; > auth_basic "Restricted"; > auth_basic_user_file /etc/nginx/.htpasswd; > } > } > 3> Add the insecure registry to /etc/sysconfig/docker & restart docker > service > OPTIONS='--insecure-registry 172.30.0.0/16 --insecure-registry 172.17.0.0/16 > --selinux-enabled' > 2.Create a imagestreamimport > # oc create -f isi > { > "kind": "ImageStreamImport", > "apiVersion": "v1", > "metadata": { > "name": "bc" > }, > "spec":{ > "import": true, > "repository": { > "from": {"kind": "DockerImage", "name": "test.com:80/test/busybox"}, > "importPolicy": {"insecure": true } > }} > } > 3. Check the imagestream & check the master log (loglevel=5) > > Actual results: > 3. > 1> # oc describe is bc > > Name: bc > Created: 34 minutes ago > Labels: <none> > Annotations: > openshift.io/image.dockerRepositoryCheck=2016-01-28T05:52:13Z > Docker Pull Spec: <none> > Tags: <none> > > 2> > {imagestreamimports /v1, Kind=ImageStreamImport 0x45b0840 0xc2081b0880 > 0xc20810a650 {}} > I0128 00:41:38.507992 20064 importer.go:684] Falling back to an HTTP check > for an insecure registry {https <nil> test.com:80 }: Get > https://test.com:80/v2/: tls: o > versized record received with length 20527 > I0128 00:41:38.514300 20064 credentials.go:108] Unable to find a secret to > match http://test.com:80/v2/test/busybox/tags/list > (test.com:80/v2/test/busybox/tags/list) > I0128 00:41:38.514324 20064 importer.go:348] unable to access tags for > repository > &importer.importRepository{Ref:api.DockerImageReference{Registry:"test.com: > 80", Name > space:"test", Name:"busybox", Tag:"", ID:""}, > Registry:(*url.URL)(0xc20f0dcb80), Name:"test/busybox", Insecure:true, > Tags:[]importer.importTag(nil), Digests:[]importer. > importDigest(nil), MaximumTags:5, AdditionalTags:[]string(nil), > Err:error(nil)}: &url.Error{Op:"Get", > URL:"http://test.com:80/v2/test/busybox/tags/list", Err:(*errors.e > rrorString)(0xc20dece750)} > I0128 00:41:38.526301 20064 image_change_controller.go:47] Build image > change controller detected ImageStream change > > > > Expected results: > Should work well > > Additional info: Checked with devenv-rhel7_3325, still can reproduce this.
I added a test but can't recreate this locally - can you provide the updated errors so I can see the output?
Checked with devenv-rhel7-3422, still can reproduce this issue. The master log: (loglevel=5) I0215 00:32:14.569221 22382 importer.go:308] importing remote Docker repository registry=https://test.com:80 repository=wjiang/busybox insecure=true I0215 00:32:14.569790 22382 importer.go:695] Falling back to an HTTP check for an insecure registry {https <nil> test.com:80 }: Get https://test.com:80/v2/: tls: oversized record received with length 20527 I0215 00:32:14.581835 22382 credentials.go:131] Unable to find a secret to match http://test.com:80/v2/wjiang/busybox/tags/list (test.com:80/v2/wjiang/busybox/tags/list) I0215 00:32:14.581857 22382 importer.go:352] unable to access tags for repository &importer.importRepository{Ref:api.DockerImageReference{Registry:"test.com:80", Namespace:"wjiang", Name:"busybox", Tag:"", ID: ""}, Registry:(*url.URL)(0xc209374700), Name:"wjiang/busybox", Insecure:true, Tags:[]importer.importTag(nil), Digests:[]importer.importDigest(nil), MaximumTags:5, AdditionalTags:[]string(nil), Err:error(nil)}: & url.Error{Op:"Get", URL:"http://test.com:80/v2/wjiang/busybox/tags/list", Err:(*errors.errorString)(0xc20de0d150)} I0215 00:32:14.582228 22382 rest.go:241] create new stream: &api.ImageStream{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"tete", GenerateName:"", Namespace:"wjiang", S elfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriod Seconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"openshift.io/image.dockerRepositoryCheck":"2016-02-15T05:32:14Z"}}, Spec:api.ImageStreamSpec{DockerImageRepository:"", Tags:ma p[string]api.TagReference(nil)}, Status:api.ImageStreamStatus{DockerImageRepository:"", Tags:map[string]api.TagEventList(nil)}} I0215 00:32:14.591703 22382 image_change_controller.go:47] Build image change controller detected ImageStream change I0215 00:32:14.593420 22382 controller.go:89] Updated all DeploymentConfigs for trigger on ImageStream wjiang/tete The reproduced steps is: 1. Setup a private docker registry 1> docker run -d registry:2 2> docker run -d -v /nginx.conf:/etc/nginx/conf.d/registry.conf -v /htpasswd:/etc/nginx/.htpasswd nginx # cat nginx.conf server { listen 80; server_name test.com; add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always; location / { proxy_pass http://172.17.0.1:5000; auth_basic "Restricted"; auth_basic_user_file /etc/nginx/.htpasswd; } } 3> Add the insecure registry to /etc/sysconfig/docker & restart docker service OPTIONS='--insecure-registry 172.30.0.0/16 --insecure-registry 172.17.0.0/16 --selinux-enabled' 2.Create a imagestreamimport # oc create -f isi { "kind": "ImageStreamImport", "apiVersion": "v1", "metadata": { "name": "bc" }, "spec":{ "import": true, "repository": { "from": {"kind": "DockerImage", "name": "test.com:80/test/busybox"}, "importPolicy": {"insecure": true } }} } 3. Check the imagestream & check the master log (loglevel=5)
Ah, this is a known bug (you used "test.com:80" in your dockercfg). Try updating your secret to use either "test.com" or "http://test.com:80". The issue is https://github.com/kubernetes/kubernetes/issues/20667
Checked with devenv-rhel7_3433, and only work when using "http://test.com:80". # oc secrets new-dockercfg test3 --docker-server=http://test.com:80 --docker-username=test --docker-password=test --docker-email=test
(In reply to weiwei jiang from comment #12) > Checked with devenv-rhel7_3433, and only work when using > "http://test.com:80". > > # oc secrets new-dockercfg test3 --docker-server=http://test.com:80 > --docker-username=test --docker-password=test --docker-email=test Are you saying that this worked after changing the configuration as suggested or are you waiting for the next rebase/cherry-pick in order to test the referenced issue which looks like it has been merged?
(In reply to Paul Weil from comment #13) > (In reply to weiwei jiang from comment #12) > > Checked with devenv-rhel7_3433, and only work when using > > "http://test.com:80". > > > > # oc secrets new-dockercfg test3 --docker-server=http://test.com:80 > > --docker-username=test --docker-password=test --docker-email=test > > Are you saying that this worked after changing the configuration as > suggested or are you waiting for the next rebase/cherry-pick in order to > test the referenced issue which looks like it has been merged? This worked after changing the configuration as suggested. and still wait for next rebase to have a try on this
Landed in the rebase.
Checked with devenv-rhel7_3734, https://github.com/kubernetes/kubernetes/pull/21286 still not in the code.
This has been merged to master as of the rebase from 20160316. So if it is not working, then it is broken.
Checked with devenv-rhel7_3839, and found it work now. I0325 06:58:27.963400 11696 credentials.go:150] Found secret to match http://test.com:80/v2/test/busybox/tags/list (test.com:80/test/busybox/tags/list):