Description of problem: images are missing: jenkins:2 and postgresql:9.6 Both is tags are available in ocp 3.9. Version-Release number of selected component (if applicable): # yum list installed | grep openshift atomic-openshift.x86_64 3.10.0-0.30.0.git.0.0b0a7a3.el7 How reproducible: Always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: $ oc get imagestreamtag -n openshift | grep postgresql postgresql:9.5 docker-registry.default.svc:5000/openshift/postgresql@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 3 hours ago sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 $ oc get imagestreamtag -n openshift | grep jenkins jenkins:1 docker-registry.default.svc:5000/openshift/jenkins@sha256:115c0685e1d9b3bcf3d4cd77d9c8f16fb0f5cf2f66517bedfc8ad2b18aef821e 3 hours ago sha256:115c0685e1d9b3bcf3d4cd77d9c8f16fb0f5cf2f66517bedfc8ad2b18aef821e ================= $ oc describe is -n openshift postgresql Name: postgresql Namespace: openshift Created: 3 hours ago Labels: <none> Annotations: openshift.io/display-name=PostgreSQL openshift.io/image.dockerRepositoryCheck=2018-04-30T12:06:14Z Docker Pull Spec: docker-registry.default.svc:5000/openshift/postgresql Image Lookup: local=false Unique Images: 1 Tags: 5 9.6 (latest) tagged from registry.access.redhat.com/rhscl/postgresql-96-rhel7:latest prefer registry pullthrough when referencing this tag Provides a PostgreSQL 9.6 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md. Tags: database, postgresql ! error: Import failed (InternalError): Internal error occurred: error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don't have permission to access \"http://access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/rhscl/1/containers/registry/rhscl/postgresql-96-rhel7/manifests/list/latest\" on this server.<P>\nReference #18.f5b31bb8.1525089974.270fe61\n</BODY>\n</HTML>\n" 3 hours ago 9.5 tagged from registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest prefer registry pullthrough when referencing this tag Provides a PostgreSQL 9.5 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.5/README.md. Tags: database, postgresql * registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 3 hours ago ========== $ oc describe is -n openshift jenkins Name: jenkins Namespace: openshift Created: 3 hours ago Labels: <none> Annotations: openshift.io/display-name=Jenkins openshift.io/image.dockerRepositoryCheck=2018-04-30T12:06:11Z Docker Pull Spec: docker-registry.default.svc:5000/openshift/jenkins Image Lookup: local=false Unique Images: 1 Tags: 3 1 tagged from registry.access.redhat.com/openshift3/jenkins-1-rhel7:latest prefer registry pullthrough when referencing this tag Provides a Jenkins 1.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md. Tags: hidden, jenkins * registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:115c0685e1d9b3bcf3d4cd77d9c8f16fb0f5cf2f66517bedfc8ad2b18aef821e 3 hours ago 2 (latest) tagged from registry.access.redhat.com/openshift3/jenkins-2-rhel7:v3.10 prefer registry pullthrough when referencing this tag Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md. Tags: jenkins ! error: Import failed (InternalError): Internal error occurred: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\"" 3 hours ago
The missing jenkins image is not unexpected, as the v3.10 images have not been published yet. We have to update our imagestreamtag refs in advance of GA, so they will refer to images that don't exist until GA. For the missing postgres image I am transferring this to the SCL team, i don't know if they expected their image to be published already or not. If not, we should probably revert the imagestreamtag.
Actually I see the postgresql 9.6 image does exist and I was able to import it using the imagestream at https://raw.githubusercontent.com/sclorg/postgresql-container/master/imagestreams/postgresql-rhel7.json Hongkai can you still recreate this and if so can you supply the output of "oc get is postgresql -n openshift -o yaml"?
$ oc get is postgresql -n openshift -o yaml apiVersion: image.openshift.io/v1 kind: ImageStream metadata: annotations: openshift.io/display-name: PostgreSQL openshift.io/image.dockerRepositoryCheck: 2018-04-30T12:06:14Z creationTimestamp: 2018-04-30T12:06:09Z generation: 2 name: postgresql namespace: openshift resourceVersion: "1226" selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/postgresql uid: de22f5a3-4c6e-11e8-92e5-02741d403af6 spec: lookupPolicy: local: false tags: - annotations: description: Provides a PostgreSQL 9.2 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.2/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL 9.2 openshift.io/provider-display-name: Red Hat, Inc. tags: hidden,postgresql version: "9.2" from: kind: DockerImage name: registry.access.redhat.com/openshift3/postgresql-92-rhel7:latest generation: 2 importPolicy: {} name: "9.2" referencePolicy: type: Local - annotations: description: Provides a PostgreSQL 9.4 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.4/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL 9.4 openshift.io/provider-display-name: Red Hat, Inc. tags: hidden,database,postgresql version: "9.4" from: kind: DockerImage name: registry.access.redhat.com/rhscl/postgresql-94-rhel7:latest generation: 2 importPolicy: {} name: "9.4" referencePolicy: type: Local - annotations: description: Provides a PostgreSQL 9.5 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.5/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL 9.5 openshift.io/provider-display-name: Red Hat, Inc. tags: database,postgresql version: "9.5" from: kind: DockerImage name: registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest generation: 2 importPolicy: {} name: "9.5" referencePolicy: type: Local - annotations: description: Provides a PostgreSQL 9.6 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL (Ephemeral) 9.6 openshift.io/provider-display-name: Red Hat, Inc. tags: database,postgresql version: "9.6" from: kind: DockerImage name: registry.access.redhat.com/rhscl/postgresql-96-rhel7:latest generation: 2 importPolicy: {} name: "9.6" referencePolicy: type: Local - annotations: description: |- Provides a PostgreSQL database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md. WARNING: By selecting this tag, your application will automatically update to use the latest version of PostgreSQL available on OpenShift, including major versions updates. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL (Latest) openshift.io/provider-display-name: Red Hat, Inc. tags: database,postgresql from: kind: ImageStreamTag name: "9.6" generation: 1 importPolicy: {} name: latest referencePolicy: type: Local status: dockerImageRepository: docker-registry.default.svc:5000/openshift/postgresql tags: - conditions: - generation: 2 lastTransitionTime: 2018-04-30T12:06:14Z message: 'Internal error occurred: error parsing HTTP 403 response body: invalid character ''<'' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don''t have permission to access \"http://access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/ose/3/containers/registry/openshift3/postgresql-92-rhel7/manifests/1/latest\" on this server.<P>\nReference #18.f5b31bb8.1525089974.270fef7\n</BODY>\n</HTML>\n"' reason: InternalError status: "False" type: ImportSuccess items: null tag: "9.2" - conditions: - generation: 2 lastTransitionTime: 2018-04-30T12:06:14Z message: 'Internal error occurred: error parsing HTTP 403 response body: invalid character ''<'' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don''t have permission to access \"http://access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/rhscl/1/containers/registry/rhscl/postgresql-94-rhel7/manifests/list/latest\" on this server.<P>\nReference #18.f5b31bb8.1525089974.270ffb2\n</BODY>\n</HTML>\n"' reason: InternalError status: "False" type: ImportSuccess items: null tag: "9.4" - items: - created: 2018-04-30T12:06:14Z dockerImageReference: registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 generation: 2 image: sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 tag: "9.5" - conditions: - generation: 2 lastTransitionTime: 2018-04-30T12:06:14Z message: 'Internal error occurred: error parsing HTTP 403 response body: invalid character ''<'' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don''t have permission to access \"http://access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/rhscl/1/containers/registry/rhscl/postgresql-96-rhel7/manifests/list/latest\" on this server.<P>\nReference #18.f5b31bb8.1525089974.270fe61\n</BODY>\n</HTML>\n"' reason: InternalError status: "False" type: ImportSuccess items: null tag: "9.6"
Is that a recreate or the yaml from the original import failure?
Sorry, Ben. I just understood what you mean by "REcreate" ... I never created it manually. The content in Comment 4 is from the original import failure. This is what I did now: oc delete imagestreams/postgresql -n openshift oc create -f https://raw.githubusercontent.com/sclorg/postgresql-container/master/imagestreams/postgresql-rhel7.json -n openshift Do I always have to do this manual step in order to use 9.6 postgresql? =============================== verify: $ oc get imagestreamtag -n openshift | grep postgre postgresql:9.2 docker-registry.default.svc:5000/openshift/postgresql@sha256:323755143c973ed639e6bcd7f261e934ee65cb4137624f449195aabc87419229 4 minutes ago sha256:323755143c973ed639e6bcd7f261e934ee65cb4137624f449195aabc87419229 postgresql:9.4 docker-registry.default.svc:5000/openshift/postgresql@sha256:3bc37b1d419398c0411d25cbb0d0d717a621105836073cad9f3ee29ffed02232 4 minutes ago sha256:3bc37b1d419398c0411d25cbb0d0d717a621105836073cad9f3ee29ffed02232 postgresql:9.5 docker-registry.default.svc:5000/openshift/postgresql@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 4 minutes ago sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 postgresql:9.6 docker-registry.default.svc:5000/openshift/postgresql@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 4 minutes ago sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 postgresql:latest docker-registry.default.svc:5000/openshift/postgresql@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 4 minutes ago sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 $ oc get is postgresql -n openshift -o yaml apiVersion: image.openshift.io/v1 kind: ImageStream metadata: annotations: openshift.io/display-name: PostgreSQL openshift.io/image.dockerRepositoryCheck: 2018-04-30T17:44:06Z creationTimestamp: 2018-04-30T17:44:02Z generation: 2 name: postgresql namespace: openshift resourceVersion: "61366" selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/postgresql uid: 11b03587-4c9e-11e8-92e5-02741d403af6 spec: lookupPolicy: local: false tags: - annotations: description: Provides a PostgreSQL 9.2 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.2/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL 9.2 openshift.io/provider-display-name: Red Hat, Inc. tags: hidden,postgresql version: "9.2" from: kind: DockerImage name: registry.access.redhat.com/openshift3/postgresql-92-rhel7:latest generation: 2 importPolicy: {} name: "9.2" referencePolicy: type: Local - annotations: description: Provides a PostgreSQL 9.4 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.4/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL 9.4 openshift.io/provider-display-name: Red Hat, Inc. tags: hidden,database,postgresql version: "9.4" from: kind: DockerImage name: registry.access.redhat.com/rhscl/postgresql-94-rhel7:latest generation: 2 importPolicy: {} name: "9.4" referencePolicy: type: Local - annotations: description: Provides a PostgreSQL 9.5 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.5/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL 9.5 openshift.io/provider-display-name: Red Hat, Inc. tags: database,postgresql version: "9.5" from: kind: DockerImage name: registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest generation: 2 importPolicy: {} name: "9.5" referencePolicy: type: Local - annotations: description: Provides a PostgreSQL 9.6 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL (Ephemeral) 9.6 openshift.io/provider-display-name: Red Hat, Inc. tags: database,postgresql version: "9.6" from: kind: DockerImage name: registry.access.redhat.com/rhscl/postgresql-96-rhel7:latest generation: 2 importPolicy: {} name: "9.6" referencePolicy: type: Local - annotations: description: |- Provides a PostgreSQL database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md. WARNING: By selecting this tag, your application will automatically update to use the latest version of PostgreSQL available on OpenShift, including major versions updates. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL (Latest) openshift.io/provider-display-name: Red Hat, Inc. tags: database,postgresql from: kind: ImageStreamTag name: "9.6" generation: 1 importPolicy: {} name: latest referencePolicy: type: Local status: dockerImageRepository: docker-registry.default.svc:5000/openshift/postgresql tags: - items: - created: 2018-04-30T17:44:06Z dockerImageReference: registry.access.redhat.com/openshift3/postgresql-92-rhel7@sha256:323755143c973ed639e6bcd7f261e934ee65cb4137624f449195aabc87419229 generation: 2 image: sha256:323755143c973ed639e6bcd7f261e934ee65cb4137624f449195aabc87419229 tag: "9.2" - items: - created: 2018-04-30T17:44:06Z dockerImageReference: registry.access.redhat.com/rhscl/postgresql-94-rhel7@sha256:3bc37b1d419398c0411d25cbb0d0d717a621105836073cad9f3ee29ffed02232 generation: 2 image: sha256:3bc37b1d419398c0411d25cbb0d0d717a621105836073cad9f3ee29ffed02232 tag: "9.4" - items: - created: 2018-04-30T17:44:06Z dockerImageReference: registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 generation: 2 image: sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 tag: "9.5" - items: - created: 2018-04-30T17:44:06Z dockerImageReference: registry.access.redhat.com/rhscl/postgresql-96-rhel7@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 generation: 2 image: sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 tag: "9.6" - items: - created: 2018-04-30T17:44:06Z dockerImageReference: registry.access.redhat.com/rhscl/postgresql-96-rhel7@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 generation: 2 image: sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 tag: latest
No, it should have imported correctly in the first place, like all the other tags, but i'm wondering if you managed to do your original install at a time when the tag was not available or there were registry issues that caused the import failure. There should be no difference in code between what you just did manually, and what happens during a normal cluster install. So i guess i'd ask if you're seeing this issue in other cluster installs, if not i'd suggest we can close it as not a bug.
I have seen the missing 96 version at least once before. This is from my teammate's cluster (also 3.10 ocp): Good thing is 96 and 95 tags are there but it is not as pretty as the one in Comment 6 (also has 92 and 94). I guess it is not stable which ones will be available then. Probably we want to know why this happens in the status. "message: 'Internal error occurred: error parsing HTTP 403 response body: invalid" ====================== root@ip-172-31-61-35: ~/svt/openshift_scalability # oc get imagestreamtag -n openshift | grep postgre postgresql:9.5 docker-registry.default.svc:5000/openshift/postgresql@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 4 hours ago sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 postgresql:9.6 docker-registry.default.svc:5000/openshift/postgresql@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 4 hours ago sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 postgresql:latest docker-registry.default.svc:5000/openshift/postgresql@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 4 hours ago sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 root@ip-172-31-61-35: ~/svt/openshift_scalability # oc get is postgresql -n openshift -o yaml apiVersion: image.openshift.io/v1 kind: ImageStream metadata: annotations: openshift.io/display-name: PostgreSQL openshift.io/image.dockerRepositoryCheck: 2018-04-30T14:52:27Z creationTimestamp: 2018-04-30T14:52:21Z generation: 2 name: postgresql namespace: openshift resourceVersion: "1226" selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/postgresql uid: 164c6f31-4c86-11e8-a620-02701b2ae108 spec: lookupPolicy: local: false tags: - annotations: description: Provides a PostgreSQL 9.2 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.2/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL 9.2 openshift.io/provider-display-name: Red Hat, Inc. tags: hidden,postgresql version: "9.2" from: kind: DockerImage name: registry.access.redhat.com/openshift3/postgresql-92-rhel7:latest generation: 2 importPolicy: {} name: "9.2" referencePolicy: type: Local - annotations: description: Provides a PostgreSQL 9.4 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.4/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL 9.4 openshift.io/provider-display-name: Red Hat, Inc. tags: hidden,database,postgresql version: "9.4" from: kind: DockerImage name: registry.access.redhat.com/rhscl/postgresql-94-rhel7:latest generation: 2 importPolicy: {} name: "9.4" referencePolicy: type: Local - annotations: description: Provides a PostgreSQL 9.5 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.5/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL 9.5 openshift.io/provider-display-name: Red Hat, Inc. tags: database,postgresql version: "9.5" from: kind: DockerImage name: registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest generation: 2 [21/110] importPolicy: {} name: "9.5" referencePolicy: type: Local - annotations: description: Provides a PostgreSQL 9.6 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL (Ephemeral) 9.6 openshift.io/provider-display-name: Red Hat, Inc. tags: database,postgresql version: "9.6" from: kind: DockerImage name: registry.access.redhat.com/rhscl/postgresql-96-rhel7:latest generation: 2 importPolicy: {} name: "9.6" referencePolicy: type: Local - annotations: description: |- Provides a PostgreSQL database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md. WARNING: By selecting this tag, your application will automatically update to use the latest version of PostgreSQL available on OpenShift, including major versions updates. iconClass: icon-postgresql openshift.io/display-name: PostgreSQL (Latest) openshift.io/provider-display-name: Red Hat, Inc. tags: database,postgresql from: kind: ImageStreamTag name: "9.6" generation: 1 importPolicy: {} name: latest referencePolicy: type: Local status: dockerImageRepository: docker-registry.default.svc:5000/openshift/postgresql tags: - conditions: - generation: 2 lastTransitionTime: 2018-04-30T14:52:27Z message: 'Internal error occurred: error parsing HTTP 403 response body: invalid character ''<'' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don''t have permission to access \"http://access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/ose/3/containers/registry/openshift3/postgresql-92& #45;rhel7/manifests/1/latest\" on this server.<P>\nReference #18.f5b31bb8.1525099947.2d15a8e\n</BODY>\n</HTML>\n"' reason: InternalError status: "False" type: ImportSuccess items: null tag: "9.2" - conditions: - generation: 2 lastTransitionTime: 2018-04-30T14:52:27Z message: 'Internal error occurred: error parsing HTTP 403 response body: invalid character ''<'' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don''t have permission to access \"http://access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/rhscl/1/containers/registry/rhscl/postgresql-94- ;rhel7/manifests/list/latest\" on this server.<P>\nReference #18.f5b31bb8.1525099947.2d15ada\n</BODY>\n</HTML>\n"' reason: InternalError status: "False" type: ImportSuccess items: null tag: "9.4" - items: - created: 2018-04-30T14:52:27Z dockerImageReference: registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 generation: 2 image: sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49 tag: "9.5" - items: - created: 2018-04-30T14:52:27Z dockerImageReference: registry.access.redhat.com/rhscl/postgresql-96-rhel7@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 generation: 2 image: sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 tag: "9.6" - items: - created: 2018-04-30T14:52:27Z dockerImageReference: registry.access.redhat.com/rhscl/postgresql-96-rhel7@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 generation: 2 image: sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7 tag: latest
I have a sense this may be related to the changes we just made for pullthrough/localreference in the imagestreams. Are you seeing this in any other imagestreams other than postgres?
(and ignoring jenkins for the reasons i covered earlier)
I will check others tomorrow and update here.
Tested again with a cluster installed this morning. # yum list installed | grep openshift atomic-openshift.x86_64 3.10.0-0.31.0.git.0.9ab12dc.el7 oc get is -o yaml -n openshift > bz1573249.txt The file is uploaded here: http://file.rdu.redhat.com/~hongkliu/test_result/bz1573249/bz1573249.txt # oc get is -o yaml -n openshift | grep message | grep "Internal error occurred" | wc -l 97 Ben, I have another question: ``` I have a sense this may be related to the changes we just made for pullthrough/localreference in the imagestreams. ``` Does that mean we will have this in the future for the is tag? ``` docker-registry.default.svc:5000/openshift/postgresql@sha256:31d46b178b84cd92e1be5348f02 ``` It was "registry.access.redhat.com/rhscl/postgre..." before. Then how do we know which the original image the local one is mapping to? By sha? Thanks.
> Does that mean we will have this in the future for the is tag? yes, the status tag will contain an image spec that points to the internal registry, not the original external source. > Then how do we know which the original image the local one is mapping to? By sha? yes, unfortunately i think that's the only way to correlate the exact image. You can see what docker repository is being referenced by getting the tag, e.g: $ oc get istag ruby:2.4 -o yaml | grep -A 2 from: from: kind: DockerImage name: docker.io/centos/ruby-24-centos7:latest Thank you for the data file, i need to do some thinking about why your imports might be failing. If you run "oc import-image someimagetream" for one of the imagestreams w/ failed imports, does it resolve them successfully?
Yes, it definitely helps. root@ip-172-31-59-64: ~ # oc get is -o yaml -n openshift | grep message | grep "Internal error occurred" | wc -l 93 root@ip-172-31-59-64: ~ # oc get imagestreamtag | grep ruby ruby:2.4 registry.access.redhat.com/rhscl/ruby-24-rhel7@sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d 8 minutes ago sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d ruby:latest docker-registry.default.svc:5000/openshift/ruby@sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d 8 minutes ago sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d root@ip-172-31-59-64: ~ # root@ip-172-31-59-64: ~ # root@ip-172-31-59-64: ~ # oc import-image ruby:2.3 The import completed successfully. Name: ruby Namespace: openshift Created: 9 hours ago Labels: <none> Annotations: openshift.io/display-name=Ruby openshift.io/image.dockerRepositoryCheck=2018-05-01T20:55:34Z Docker Pull Spec: docker-registry.default.svc:5000/openshift/ruby Image Lookup: local=false Unique Images: 2 Tags: 5 2.4 (latest) tagged from registry.access.redhat.com/rhscl/ruby-24-rhel7:latest Build and run Ruby 2.4 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/blob/master/2.4/README.md. Tags: builder, ruby Supports: ruby:2.4, ruby Example Repo: https://github.com/openshift/ruby-ex.git * registry.access.redhat.com/rhscl/ruby-24-rhel7@sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d 9 minutes ago 2.3 tagged from registry.access.redhat.com/rhscl/ruby-23-rhel7:latest Build and run Ruby 2.3 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/blob/master/2.3/README.md. Tags: builder, ruby Supports: ruby:2.3, ruby Example Repo: https://github.com/openshift/ruby-ex.git * registry.access.redhat.com/rhscl/ruby-23-rhel7@sha256:306e3cf5a6a9c9323c776fbe12ac35c6f621f44d4154ce0944a844bbec93e51f Less than a second ago 2.2 tagged from registry.access.redhat.com/rhscl/ruby-22-rhel7:latest prefer registry pullthrough when referencing this tag Build and run Ruby 2.2 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.2/README.md. Tags: hidden, builder, ruby Supports: ruby:2.2, ruby Example Repo: https://github.com/openshift/ruby-ex.git ! error: Import failed (InternalError): Internal error occurred: error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don't have permission to access \"http://access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/rhscl/1/containers/registry/rhscl/ruby-22-rhel7/manifests/list/latest\" on this server.<P>\nReference #18.f5b31bb8.1525174857.3cb78b\n</BODY>\n</HTML>\n" 9 hours ago 2.0 tagged from registry.access.redhat.com/openshift3/ruby-20-rhel7:latest prefer registry pullthrough when referencing this tag Build and run Ruby 2.0 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.0/README.md. Tags: hidden, builder, ruby Supports: ruby:2.0, ruby Example Repo: https://github.com/openshift/ruby-ex.git ! error: Import failed (InternalError): Internal error occurred: error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don't have permission to access \"http://access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/ose/3/containers/registry/openshift3/ruby-20-rhel7/manifests/1/latest\" on this server.<P>\nReference #18.f5b31bb8.1525174857.3cb723\n</BODY>\n</HTML>\n" 9 hours ago Image Name: ruby:latest Docker Image: registry.access.redhat.com/rhscl/ruby-23-rhel7@sha256:306e3cf5a6a9c9323c776fbe12ac35c6f621f44d4154ce0944a844bbec93e51f Name: sha256:306e3cf5a6a9c9323c776fbe12ac35c6f621f44d4154ce0944a844bbec93e51f Created: Less than a second ago Annotations: image.openshift.io/dockerLayersOrder=ascending Image Size: 197.7MB (first layer 74.96MB, last binary layer 25.23MB) Image Created: 3 weeks ago Author: <none> Arch: amd64 Entrypoint: container-entrypoint Command: /bin/sh -c $STI_SCRIPTS_PATH/usage Working Dir: /opt/app-root/src User: 1001 Exposes Ports: 8080/tcp Docker Labels: architecture=x86_64 authoritative-source-url=registry.access.redhat.com build-date=2018-04-05T10:49:35.087100 com.redhat.build-host=osbs-cpt-010.ocp.osbs.upshift.eng.rdu2.redhat.com com.redhat.component=rh-ruby23-docker description=Ruby 2.3 available as container is a base platform for building and running various Ruby 2.3 applications and frameworks. Ruby is the interpreted scripting language for quick and easy object-oriented programming. It has many features to process text files and to do system management tasks (as in Perl). It is simple, straight-forward, and extensible. distribution-scope=public io.k8s.description=Ruby 2.3 available as container is a base platform for building and running various Ruby 2.3 applications and frameworks. Ruby is the interpreted scripting language for quick and easy object-oriented programming. It has many features to process text files and to do system management tasks (as in Perl). It is simple, straight-forward, and extensible. io.k8s.display-name=Ruby 2.3 io.openshift.expose-services=8080:http io.openshift.s2i.scripts-url=image:///usr/libexec/s2i io.openshift.tags=builder,ruby,ruby23,rh-ruby23 io.s2i.scripts-url=image:///usr/libexec/s2i maintainer=SoftwareCollections.org <sclorg> name=rhscl/ruby-23-rhel7 release=23 summary=Platform for building and running Ruby 2.3 applications url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhscl/ruby-23-rhel7/images/2.3-23 usage=s2i build https://github.com/sclorg/s2i-ruby-container.git --context-dir=2.4/test/puma-test-app/ rhscl/ruby-23-rhel7 ruby-sample-app vcs-ref=b6676600478a3dfa1e3352205d648b1c54697ecf vcs-type=git vendor=Red Hat, Inc. version=2.3 Environment: PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci SUMMARY=Platform for building and running Ruby 2.3 applications DESCRIPTION=Ruby 2.3 available as container is a base platform for building and running various Ruby 2.3 applications and frameworks. Ruby is the interpreted scripting language for quick and easy object-oriented programming. It has many features to process text files and to do system management tasks (as in Perl). It is simple, straight-forward, and extensible. STI_SCRIPTS_URL=image:///usr/libexec/s2i STI_SCRIPTS_PATH=/usr/libexec/s2i APP_ROOT=/opt/app-root HOME=/opt/app-root/src BASH_ENV=/opt/app-root/etc/scl_enable ENV=/opt/app-root/etc/scl_enable PROMPT_COMMAND=. /opt/app-root/etc/scl_enable NODEJS_SCL=rh-nodejs6 RUBY_VERSION=2.3 root@ip-172-31-59-64: ~ # oc get is -o yaml -n openshift | grep message | grep "Internal error occurred" | wc -l 92 root@ip-172-31-59-64: ~ # oc get imagestreamtag | grep ruby ruby:2.3 registry.access.redhat.com/rhscl/ruby-23-rhel7@sha256:306e3cf5a6a9c9323c776fbe12ac35c6f621f44d4154ce0944a844bbec93e51f 9 seconds ago sha256:306e3cf5a6a9c9323c776fbe12ac35c6f621f44d4154ce0944a844bbec93e51f ruby:2.4 registry.access.redhat.com/rhscl/ruby-24-rhel7@sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d 9 minutes ago sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d ruby:latest docker-registry.default.svc:5000/openshift/ruby@sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d 9 minutes ago sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d
Ben, were there any known issues w/ the registry in the last week? It seems like openshift has been getting 403ed while trying to access urls like: http://access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/ose/3/containers/registry/openshift3/ruby-20-rhel7/manifests/1/latest which i would expect to be open/not require authentication. Hongkai is this being seen on other cluster installations performed by the QE team?
Let me check it today if we still have the problem.
Tested with # yum list installed | grep openshift atomic-openshift.x86_64 3.10.0-0.32.0.git.0.2b17fd0.el7 Much much better now. # oc get is -o yaml -n openshift | grep message | grep "Internal error occurred" | wc -l 2 I guess those 2 are expected: ==================== status: dockerImageRepository: docker-registry.default.svc:5000/openshift/jboss-eap71-openshift tags: - conditions: - generation: 2 lastTransitionTime: 2018-05-08T11:57:45Z message: 'Internal error occurred: unknown: Not Found' reason: InternalError status: "False" type: ImportSuccess items: null tag: 1.0-TP dockerImageRepository: docker-registry.default.svc:5000/openshift/jenkins tags: - items: - created: 2018-05-08T11:57:19Z dockerImageReference: registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:a75802c902d7056e19cc4de6210e5312db396e638cebb8291058a4378199450a generation: 2 image: sha256:a75802c902d7056e19cc4de6210e5312db396e638cebb8291058a4378199450a tag: "1" - conditions: - generation: 2 lastTransitionTime: 2018-05-08T11:57:19Z message: 'Internal error occurred: error parsing HTTP 404 response body: invalid character ''F'' looking for beginning of value: "File not found.\""' reason: InternalError status: "False" type: ImportSuccess items: null tag: "2"
can you supply the full imagestream yaml? I want to look at those two failures slightly closer.
yes, those are both expected. i'm going to move this to ON_QA in hopes that the failures were caused by registry hiccups but please reopen it if you experience this again.
Thanks, Ben.