Bug 1573249 - images are missing: jenkins:2 and postgresql:9.6
Summary: images are missing: jenkins:2 and postgresql:9.6
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: ImageStreams
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.10.0
Assignee: Ben Parees
QA Contact: Dongbo Yan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-30 15:26 UTC by Hongkai Liu
Modified: 2018-05-09 12:37 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-09 12:37:10 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Hongkai Liu 2018-04-30 15:26:23 UTC
Description of problem:
images are missing: jenkins:2 and postgresql:9.6

Both is tags are available in ocp 3.9.

Version-Release number of selected component (if applicable):
# yum list installed | grep openshift
atomic-openshift.x86_64       3.10.0-0.30.0.git.0.0b0a7a3.el7


How reproducible: Always


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:



$ oc get imagestreamtag -n openshift | grep postgresql
postgresql:9.5   docker-registry.default.svc:5000/openshift/postgresql@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49   3 hours ago   sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49

$ oc get imagestreamtag -n openshift | grep jenkins
jenkins:1        docker-registry.default.svc:5000/openshift/jenkins@sha256:115c0685e1d9b3bcf3d4cd77d9c8f16fb0f5cf2f66517bedfc8ad2b18aef821e      3 hours ago   sha256:115c0685e1d9b3bcf3d4cd77d9c8f16fb0f5cf2f66517bedfc8ad2b18aef821e

=================

$ oc describe is -n openshift postgresql
Name:			postgresql
Namespace:		openshift
Created:		3 hours ago
Labels:			<none>
Annotations:		openshift.io/display-name=PostgreSQL
			openshift.io/image.dockerRepositoryCheck=2018-04-30T12:06:14Z
Docker Pull Spec:	docker-registry.default.svc:5000/openshift/postgresql
Image Lookup:		local=false
Unique Images:		1
Tags:			5

9.6 (latest)
  tagged from registry.access.redhat.com/rhscl/postgresql-96-rhel7:latest
    prefer registry pullthrough when referencing this tag

  Provides a PostgreSQL 9.6 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md.
  Tags: database, postgresql

  ! error: Import failed (InternalError): Internal error occurred: error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don't have permission to access \"http&#58;&#47;&#47;access&#46;redhat&#46;com&#47;webassets&#47;docker&#47;content&#47;dist&#47;rhel&#47;server&#47;7&#47;7Server&#47;multiarch&#47;rhscl&#47;1&#47;containers&#47;registry&#47;rhscl&#47;postgresql&#45;96&#45;rhel7&#47;manifests&#47;list&#47;latest\" on this server.<P>\nReference&#32;&#35;18&#46;f5b31bb8&#46;1525089974&#46;270fe61\n</BODY>\n</HTML>\n"
      3 hours ago

9.5
  tagged from registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest
    prefer registry pullthrough when referencing this tag

  Provides a PostgreSQL 9.5 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.5/README.md.
  Tags: database, postgresql

  * registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49
      3 hours ago


==========

$ oc describe is -n openshift jenkins
Name:			jenkins
Namespace:		openshift
Created:		3 hours ago
Labels:			<none>
Annotations:		openshift.io/display-name=Jenkins
			openshift.io/image.dockerRepositoryCheck=2018-04-30T12:06:11Z
Docker Pull Spec:	docker-registry.default.svc:5000/openshift/jenkins
Image Lookup:		local=false
Unique Images:		1
Tags:			3

1
  tagged from registry.access.redhat.com/openshift3/jenkins-1-rhel7:latest
    prefer registry pullthrough when referencing this tag

  Provides a Jenkins 1.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.
  Tags: hidden, jenkins

  * registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:115c0685e1d9b3bcf3d4cd77d9c8f16fb0f5cf2f66517bedfc8ad2b18aef821e
      3 hours ago

2 (latest)
  tagged from registry.access.redhat.com/openshift3/jenkins-2-rhel7:v3.10
    prefer registry pullthrough when referencing this tag

  Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.
  Tags: jenkins

  ! error: Import failed (InternalError): Internal error occurred: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\""
      3 hours ago

Comment 1 Ben Parees 2018-04-30 16:27:33 UTC
The missing jenkins image is not unexpected, as the v3.10 images have not been published yet.  We have to update our imagestreamtag refs in advance of GA, so they will refer to images that don't exist until GA.

For the missing postgres image I am transferring this to the SCL team, i don't know if they expected their image to be published already or not.  If not, we should probably revert the imagestreamtag.

Comment 3 Ben Parees 2018-04-30 16:34:46 UTC
Actually I see the postgresql 9.6 image does exist and I was able to import it using the imagestream at https://raw.githubusercontent.com/sclorg/postgresql-container/master/imagestreams/postgresql-rhel7.json

Hongkai can you still recreate this and if so can you supply the output of "oc get is postgresql -n openshift -o yaml"?

Comment 4 Hongkai Liu 2018-04-30 17:20:39 UTC


$ oc get is postgresql -n openshift -o yaml
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/display-name: PostgreSQL
    openshift.io/image.dockerRepositoryCheck: 2018-04-30T12:06:14Z
  creationTimestamp: 2018-04-30T12:06:09Z
  generation: 2
  name: postgresql
  namespace: openshift
  resourceVersion: "1226"
  selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/postgresql
  uid: de22f5a3-4c6e-11e8-92e5-02741d403af6
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations:
      description: Provides a PostgreSQL 9.2 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.2/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL 9.2
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: hidden,postgresql
      version: "9.2"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/openshift3/postgresql-92-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "9.2"
    referencePolicy:
      type: Local
  - annotations:
      description: Provides a PostgreSQL 9.4 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.4/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL 9.4
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: hidden,database,postgresql
      version: "9.4"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/rhscl/postgresql-94-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "9.4"
    referencePolicy:
      type: Local
  - annotations:
      description: Provides a PostgreSQL 9.5 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.5/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL 9.5
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: database,postgresql
      version: "9.5"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "9.5"
    referencePolicy:
      type: Local
  - annotations:
      description: Provides a PostgreSQL 9.6 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL (Ephemeral) 9.6
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: database,postgresql
      version: "9.6"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/rhscl/postgresql-96-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "9.6"
    referencePolicy:
      type: Local
  - annotations:
      description: |-
        Provides a PostgreSQL database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md.

        WARNING: By selecting this tag, your application will automatically update to use the latest version of PostgreSQL available on OpenShift, including major versions updates.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL (Latest)
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: database,postgresql
    from:
      kind: ImageStreamTag
      name: "9.6"
    generation: 1
    importPolicy: {}
    name: latest
    referencePolicy:
      type: Local
status:
  dockerImageRepository: docker-registry.default.svc:5000/openshift/postgresql
  tags:
  - conditions:
    - generation: 2
      lastTransitionTime: 2018-04-30T12:06:14Z
      message: 'Internal error occurred: error parsing HTTP 403 response body: invalid
        character ''<'' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access
        Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don''t have
        permission to access \"http&#58;&#47;&#47;access&#46;redhat&#46;com&#47;webassets&#47;docker&#47;content&#47;dist&#47;rhel&#47;server&#47;7&#47;7Server&#47;multiarch&#47;ose&#47;3&#47;containers&#47;registry&#47;openshift3&#47;postgresql&#45;92&#45;rhel7&#47;manifests&#47;1&#47;latest\"
        on this server.<P>\nReference&#32;&#35;18&#46;f5b31bb8&#46;1525089974&#46;270fef7\n</BODY>\n</HTML>\n"'
      reason: InternalError
      status: "False"
      type: ImportSuccess
    items: null
    tag: "9.2"
  - conditions:
    - generation: 2
      lastTransitionTime: 2018-04-30T12:06:14Z
      message: 'Internal error occurred: error parsing HTTP 403 response body: invalid
        character ''<'' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access
        Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don''t have
        permission to access \"http&#58;&#47;&#47;access&#46;redhat&#46;com&#47;webassets&#47;docker&#47;content&#47;dist&#47;rhel&#47;server&#47;7&#47;7Server&#47;multiarch&#47;rhscl&#47;1&#47;containers&#47;registry&#47;rhscl&#47;postgresql&#45;94&#45;rhel7&#47;manifests&#47;list&#47;latest\"
        on this server.<P>\nReference&#32;&#35;18&#46;f5b31bb8&#46;1525089974&#46;270ffb2\n</BODY>\n</HTML>\n"'
      reason: InternalError
      status: "False"
      type: ImportSuccess
    items: null
    tag: "9.4"
  - items:
    - created: 2018-04-30T12:06:14Z
      dockerImageReference: registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49
      generation: 2
      image: sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49
    tag: "9.5"
  - conditions:
    - generation: 2
      lastTransitionTime: 2018-04-30T12:06:14Z
      message: 'Internal error occurred: error parsing HTTP 403 response body: invalid
        character ''<'' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access
        Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don''t have
        permission to access \"http&#58;&#47;&#47;access&#46;redhat&#46;com&#47;webassets&#47;docker&#47;content&#47;dist&#47;rhel&#47;server&#47;7&#47;7Server&#47;multiarch&#47;rhscl&#47;1&#47;containers&#47;registry&#47;rhscl&#47;postgresql&#45;96&#45;rhel7&#47;manifests&#47;list&#47;latest\"
        on this server.<P>\nReference&#32;&#35;18&#46;f5b31bb8&#46;1525089974&#46;270fe61\n</BODY>\n</HTML>\n"'
      reason: InternalError
      status: "False"
      type: ImportSuccess
    items: null
    tag: "9.6"

Comment 5 Ben Parees 2018-04-30 17:23:28 UTC
Is that a recreate or the yaml from the original import failure?

Comment 6 Hongkai Liu 2018-04-30 17:51:39 UTC
Sorry, Ben.

I just understood what you mean by "REcreate" ... I never created it manually.
The content in Comment 4 is from the original import failure.

This is what I did now:

oc delete imagestreams/postgresql -n openshift
oc create -f https://raw.githubusercontent.com/sclorg/postgresql-container/master/imagestreams/postgresql-rhel7.json -n openshift

Do I always have to do this manual step in order to use 9.6 postgresql?

===============================
verify:
$ oc get imagestreamtag -n openshift | grep postgre
postgresql:9.2      docker-registry.default.svc:5000/openshift/postgresql@sha256:323755143c973ed639e6bcd7f261e934ee65cb4137624f449195aabc87419229   4 minutes ago   sha256:323755143c973ed639e6bcd7f261e934ee65cb4137624f449195aabc87419229
postgresql:9.4      docker-registry.default.svc:5000/openshift/postgresql@sha256:3bc37b1d419398c0411d25cbb0d0d717a621105836073cad9f3ee29ffed02232   4 minutes ago   sha256:3bc37b1d419398c0411d25cbb0d0d717a621105836073cad9f3ee29ffed02232
postgresql:9.5      docker-registry.default.svc:5000/openshift/postgresql@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49   4 minutes ago   sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49
postgresql:9.6      docker-registry.default.svc:5000/openshift/postgresql@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7   4 minutes ago   sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7
postgresql:latest   docker-registry.default.svc:5000/openshift/postgresql@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7   4 minutes ago   sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7



$ oc get is postgresql -n openshift -o yaml
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/display-name: PostgreSQL
    openshift.io/image.dockerRepositoryCheck: 2018-04-30T17:44:06Z
  creationTimestamp: 2018-04-30T17:44:02Z
  generation: 2
  name: postgresql
  namespace: openshift
  resourceVersion: "61366"
  selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/postgresql
  uid: 11b03587-4c9e-11e8-92e5-02741d403af6
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations:
      description: Provides a PostgreSQL 9.2 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.2/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL 9.2
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: hidden,postgresql
      version: "9.2"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/openshift3/postgresql-92-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "9.2"
    referencePolicy:
      type: Local
  - annotations:
      description: Provides a PostgreSQL 9.4 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.4/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL 9.4
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: hidden,database,postgresql
      version: "9.4"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/rhscl/postgresql-94-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "9.4"
    referencePolicy:
      type: Local
  - annotations:
      description: Provides a PostgreSQL 9.5 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.5/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL 9.5
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: database,postgresql
      version: "9.5"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "9.5"
    referencePolicy:
      type: Local
  - annotations:
      description: Provides a PostgreSQL 9.6 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL (Ephemeral) 9.6
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: database,postgresql
      version: "9.6"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/rhscl/postgresql-96-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "9.6"
    referencePolicy:
      type: Local
  - annotations:
      description: |-
        Provides a PostgreSQL database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md.

        WARNING: By selecting this tag, your application will automatically update to use the latest version of PostgreSQL available on OpenShift, including major versions updates.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL (Latest)
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: database,postgresql
    from:
      kind: ImageStreamTag
      name: "9.6"
    generation: 1
    importPolicy: {}
    name: latest
    referencePolicy:
      type: Local
status:
  dockerImageRepository: docker-registry.default.svc:5000/openshift/postgresql
  tags:
  - items:
    - created: 2018-04-30T17:44:06Z
      dockerImageReference: registry.access.redhat.com/openshift3/postgresql-92-rhel7@sha256:323755143c973ed639e6bcd7f261e934ee65cb4137624f449195aabc87419229
      generation: 2
      image: sha256:323755143c973ed639e6bcd7f261e934ee65cb4137624f449195aabc87419229
    tag: "9.2"
  - items:
    - created: 2018-04-30T17:44:06Z
      dockerImageReference: registry.access.redhat.com/rhscl/postgresql-94-rhel7@sha256:3bc37b1d419398c0411d25cbb0d0d717a621105836073cad9f3ee29ffed02232
      generation: 2
      image: sha256:3bc37b1d419398c0411d25cbb0d0d717a621105836073cad9f3ee29ffed02232
    tag: "9.4"
  - items:
    - created: 2018-04-30T17:44:06Z
      dockerImageReference: registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49
      generation: 2
      image: sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49
    tag: "9.5"
  - items:
    - created: 2018-04-30T17:44:06Z
      dockerImageReference: registry.access.redhat.com/rhscl/postgresql-96-rhel7@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7
      generation: 2
      image: sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7
    tag: "9.6"
  - items:
    - created: 2018-04-30T17:44:06Z
      dockerImageReference: registry.access.redhat.com/rhscl/postgresql-96-rhel7@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7
      generation: 2
      image: sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7
    tag: latest

Comment 7 Ben Parees 2018-04-30 18:37:29 UTC
No, it should have imported correctly in the first place, like all the other tags, but i'm wondering if you managed to do your original install at a time when the tag was not available or there were registry issues that caused the import failure.

There should be no difference in code between what you just did manually, and what happens during a normal cluster install.

So i guess i'd ask if you're seeing this issue in other cluster installs, if not i'd suggest we can close it as not a bug.

Comment 8 Hongkai Liu 2018-04-30 19:16:25 UTC
I have seen the missing 96 version at least once before.

This is from my teammate's cluster (also 3.10 ocp):
Good thing is 96 and 95 tags are there but it is not as pretty as the one in Comment 6 (also has 92 and 94).

I guess it is not stable which ones will be available then.
Probably we want to know why this happens in the status.

"message: 'Internal error occurred: error parsing HTTP 403 response body: invalid"


======================

root@ip-172-31-61-35: ~/svt/openshift_scalability # oc get imagestreamtag -n openshift | grep postgre
postgresql:9.5      docker-registry.default.svc:5000/openshift/postgresql@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49   4 hours ago   sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49
postgresql:9.6      docker-registry.default.svc:5000/openshift/postgresql@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7   4 hours ago   sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7
postgresql:latest   docker-registry.default.svc:5000/openshift/postgresql@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7   4 hours ago   sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7
root@ip-172-31-61-35: ~/svt/openshift_scalability # oc get is postgresql -n openshift -o yaml
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/display-name: PostgreSQL
    openshift.io/image.dockerRepositoryCheck: 2018-04-30T14:52:27Z
  creationTimestamp: 2018-04-30T14:52:21Z
  generation: 2
  name: postgresql
  namespace: openshift
  resourceVersion: "1226"
  selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/postgresql
  uid: 164c6f31-4c86-11e8-a620-02701b2ae108
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations:
      description: Provides a PostgreSQL 9.2 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.2/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL 9.2
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: hidden,postgresql
      version: "9.2"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/openshift3/postgresql-92-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "9.2"
    referencePolicy:
      type: Local
  - annotations:
      description: Provides a PostgreSQL 9.4 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.4/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL 9.4
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: hidden,database,postgresql
      version: "9.4"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/rhscl/postgresql-94-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "9.4"
    referencePolicy:
      type: Local
  - annotations:
      description: Provides a PostgreSQL 9.5 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.5/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL 9.5
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: database,postgresql
      version: "9.5"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/rhscl/postgresql-95-rhel7:latest
    generation: 2                                                                                                                                                                                                                                    [21/110]
    importPolicy: {}
    name: "9.5"
    referencePolicy:
      type: Local
  - annotations:
      description: Provides a PostgreSQL 9.6 database on RHEL 7. For more information
        about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL (Ephemeral) 9.6
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: database,postgresql
      version: "9.6"
    from:
      kind: DockerImage
      name: registry.access.redhat.com/rhscl/postgresql-96-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "9.6"
    referencePolicy:
      type: Local
  - annotations:
      description: |-
        Provides a PostgreSQL database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/tree/master/9.6/README.md.

        WARNING: By selecting this tag, your application will automatically update to use the latest version of PostgreSQL available on OpenShift, including major versions updates.
      iconClass: icon-postgresql
      openshift.io/display-name: PostgreSQL (Latest)
      openshift.io/provider-display-name: Red Hat, Inc.
      tags: database,postgresql
    from:
      kind: ImageStreamTag
      name: "9.6"
    generation: 1
    importPolicy: {}
    name: latest
    referencePolicy:
      type: Local
status:
  dockerImageRepository: docker-registry.default.svc:5000/openshift/postgresql
  tags:
  - conditions:
    - generation: 2
      lastTransitionTime: 2018-04-30T14:52:27Z
      message: 'Internal error occurred: error parsing HTTP 403 response body: invalid
        character ''<'' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access
        Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don''t have
        permission to access \"http&#58;&#47;&#47;access&#46;redhat&#46;com&#47;webassets&#47;docker&#47;content&#47;dist&#47;rhel&#47;server&#47;7&#47;7Server&#47;multiarch&#47;ose&#47;3&#47;containers&#47;registry&#47;openshift3&#47;postgresql&#45;92&
#45;rhel7&#47;manifests&#47;1&#47;latest\"
        on this server.<P>\nReference&#32;&#35;18&#46;f5b31bb8&#46;1525099947&#46;2d15a8e\n</BODY>\n</HTML>\n"'
      reason: InternalError
      status: "False"
      type: ImportSuccess
    items: null
    tag: "9.2"
  - conditions:
    - generation: 2
      lastTransitionTime: 2018-04-30T14:52:27Z
      message: 'Internal error occurred: error parsing HTTP 403 response body: invalid
        character ''<'' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access
        Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don''t have
        permission to access \"http&#58;&#47;&#47;access&#46;redhat&#46;com&#47;webassets&#47;docker&#47;content&#47;dist&#47;rhel&#47;server&#47;7&#47;7Server&#47;multiarch&#47;rhscl&#47;1&#47;containers&#47;registry&#47;rhscl&#47;postgresql&#45;94&#45
;rhel7&#47;manifests&#47;list&#47;latest\"
        on this server.<P>\nReference&#32;&#35;18&#46;f5b31bb8&#46;1525099947&#46;2d15ada\n</BODY>\n</HTML>\n"'
      reason: InternalError
      status: "False"
      type: ImportSuccess
    items: null
    tag: "9.4"
  - items:
    - created: 2018-04-30T14:52:27Z
      dockerImageReference: registry.access.redhat.com/rhscl/postgresql-95-rhel7@sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49
      generation: 2
      image: sha256:31d46b178b84cd92e1be5348f02a75f6a39867d5a57eb099b7c864deaaf02a49
    tag: "9.5"
  - items:
    - created: 2018-04-30T14:52:27Z
      dockerImageReference: registry.access.redhat.com/rhscl/postgresql-96-rhel7@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7
      generation: 2
      image: sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7
    tag: "9.6"
  - items:
    - created: 2018-04-30T14:52:27Z
      dockerImageReference: registry.access.redhat.com/rhscl/postgresql-96-rhel7@sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7
      generation: 2
      image: sha256:2fea3ba5978862cc04ce5921462e9ca9caaec03f095c9b5c7be074358861b5c7
    tag: latest

Comment 9 Ben Parees 2018-04-30 22:11:52 UTC
I have a sense this may be related to the changes we just made for pullthrough/localreference in the imagestreams. 

Are you seeing this in any other imagestreams other than postgres?

Comment 10 Ben Parees 2018-04-30 22:12:23 UTC
(and ignoring jenkins for the reasons i covered earlier)

Comment 11 Hongkai Liu 2018-04-30 22:42:26 UTC
I will check others tomorrow and update here.

Comment 12 Hongkai Liu 2018-04-30 22:42:42 UTC
I will check others tomorrow and update here.

Comment 13 Hongkai Liu 2018-05-01 14:47:50 UTC
Tested again with a cluster installed this morning.
# yum list installed | grep openshift
atomic-openshift.x86_64       3.10.0-0.31.0.git.0.9ab12dc.el7

oc get is -o yaml -n openshift > bz1573249.txt

The file is uploaded here:
http://file.rdu.redhat.com/~hongkliu/test_result/bz1573249/bz1573249.txt


# oc get is -o yaml -n openshift | grep message | grep "Internal error occurred" | wc -l
97


Ben,
I have another question:

```
I have a sense this may be related to the changes we just made for pullthrough/localreference in the imagestreams. 
```

Does that mean we will have this in the future for the is tag?
```
docker-registry.default.svc:5000/openshift/postgresql@sha256:31d46b178b84cd92e1be5348f02
```

It was "registry.access.redhat.com/rhscl/postgre..." before.

Then how do we know which the original image the local one is mapping to? By sha?

Thanks.

Comment 14 Ben Parees 2018-05-01 17:08:58 UTC
> Does that mean we will have this in the future for the is tag?

yes, the status tag will contain an image spec that points to the internal registry, not the original external source.

> Then how do we know which the original image the local one is mapping to? By sha?


yes, unfortunately i think that's the only way to correlate the exact image.  You can see what docker repository is being referenced by getting the tag, e.g:

$ oc get istag ruby:2.4 -o yaml | grep -A 2 from: 
  from:
    kind: DockerImage
    name: docker.io/centos/ruby-24-centos7:latest

Thank you for the data file, i need to do some thinking about why your imports might be failing.

If you run "oc import-image someimagetream" for one of the imagestreams w/ failed imports, does it resolve them successfully?

Comment 15 Hongkai Liu 2018-05-01 20:57:26 UTC
Yes, it definitely helps.

root@ip-172-31-59-64: ~ # oc get is -o yaml -n openshift | grep message | grep "Internal error occurred" | wc -l
93
root@ip-172-31-59-64: ~ # oc get imagestreamtag | grep ruby
ruby:2.4                         registry.access.redhat.com/rhscl/ruby-24-rhel7@sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d                     8 minutes ago        sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d
ruby:latest                      docker-registry.default.svc:5000/openshift/ruby@sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d                    8 minutes ago        sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d
root@ip-172-31-59-64: ~ # 
root@ip-172-31-59-64: ~ # 
root@ip-172-31-59-64: ~ # oc import-image ruby:2.3
The import completed successfully.

Name:			ruby
Namespace:		openshift
Created:		9 hours ago
Labels:			<none>
Annotations:		openshift.io/display-name=Ruby
			openshift.io/image.dockerRepositoryCheck=2018-05-01T20:55:34Z
Docker Pull Spec:	docker-registry.default.svc:5000/openshift/ruby
Image Lookup:		local=false
Unique Images:		2
Tags:			5

2.4 (latest)
  tagged from registry.access.redhat.com/rhscl/ruby-24-rhel7:latest

  Build and run Ruby 2.4 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/blob/master/2.4/README.md.
  Tags: builder, ruby
  Supports: ruby:2.4, ruby
  Example Repo: https://github.com/openshift/ruby-ex.git

  * registry.access.redhat.com/rhscl/ruby-24-rhel7@sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d
      9 minutes ago

2.3
  tagged from registry.access.redhat.com/rhscl/ruby-23-rhel7:latest

  Build and run Ruby 2.3 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/blob/master/2.3/README.md.
  Tags: builder, ruby
  Supports: ruby:2.3, ruby
  Example Repo: https://github.com/openshift/ruby-ex.git

  * registry.access.redhat.com/rhscl/ruby-23-rhel7@sha256:306e3cf5a6a9c9323c776fbe12ac35c6f621f44d4154ce0944a844bbec93e51f
      Less than a second ago

2.2
  tagged from registry.access.redhat.com/rhscl/ruby-22-rhel7:latest
    prefer registry pullthrough when referencing this tag

  Build and run Ruby 2.2 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.2/README.md.
  Tags: hidden, builder, ruby
  Supports: ruby:2.2, ruby
  Example Repo: https://github.com/openshift/ruby-ex.git

  ! error: Import failed (InternalError): Internal error occurred: error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don't have permission to access \"http&#58;&#47;&#47;access&#46;redhat&#46;com&#47;webassets&#47;docker&#47;content&#47;dist&#47;rhel&#47;server&#47;7&#47;7Server&#47;multiarch&#47;rhscl&#47;1&#47;containers&#47;registry&#47;rhscl&#47;ruby&#45;22&#45;rhel7&#47;manifests&#47;list&#47;latest\" on this server.<P>\nReference&#32;&#35;18&#46;f5b31bb8&#46;1525174857&#46;3cb78b\n</BODY>\n</HTML>\n"
      9 hours ago

2.0
  tagged from registry.access.redhat.com/openshift3/ruby-20-rhel7:latest
    prefer registry pullthrough when referencing this tag

  Build and run Ruby 2.0 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.0/README.md.
  Tags: hidden, builder, ruby
  Supports: ruby:2.0, ruby
  Example Repo: https://github.com/openshift/ruby-ex.git

  ! error: Import failed (InternalError): Internal error occurred: error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don't have permission to access \"http&#58;&#47;&#47;access&#46;redhat&#46;com&#47;webassets&#47;docker&#47;content&#47;dist&#47;rhel&#47;server&#47;7&#47;7Server&#47;multiarch&#47;ose&#47;3&#47;containers&#47;registry&#47;openshift3&#47;ruby&#45;20&#45;rhel7&#47;manifests&#47;1&#47;latest\" on this server.<P>\nReference&#32;&#35;18&#46;f5b31bb8&#46;1525174857&#46;3cb723\n</BODY>\n</HTML>\n"
      9 hours ago

Image Name:	ruby:latest
Docker Image:	registry.access.redhat.com/rhscl/ruby-23-rhel7@sha256:306e3cf5a6a9c9323c776fbe12ac35c6f621f44d4154ce0944a844bbec93e51f
Name:		sha256:306e3cf5a6a9c9323c776fbe12ac35c6f621f44d4154ce0944a844bbec93e51f
Created:	Less than a second ago
Annotations:	image.openshift.io/dockerLayersOrder=ascending
Image Size:	197.7MB (first layer 74.96MB, last binary layer 25.23MB)
Image Created:	3 weeks ago
Author:		<none>
Arch:		amd64
Entrypoint:	container-entrypoint
Command:	/bin/sh -c $STI_SCRIPTS_PATH/usage
Working Dir:	/opt/app-root/src
User:		1001
Exposes Ports:	8080/tcp
Docker Labels:	architecture=x86_64
		authoritative-source-url=registry.access.redhat.com
		build-date=2018-04-05T10:49:35.087100
		com.redhat.build-host=osbs-cpt-010.ocp.osbs.upshift.eng.rdu2.redhat.com
		com.redhat.component=rh-ruby23-docker
		description=Ruby 2.3 available as container is a base platform for building and running various Ruby 2.3 applications and frameworks. Ruby is the interpreted scripting language for quick and easy object-oriented programming. It has many features to process text files and to do system management tasks (as in Perl). It is simple, straight-forward, and extensible.
		distribution-scope=public
		io.k8s.description=Ruby 2.3 available as container is a base platform for building and running various Ruby 2.3 applications and frameworks. Ruby is the interpreted scripting language for quick and easy object-oriented programming. It has many features to process text files and to do system management tasks (as in Perl). It is simple, straight-forward, and extensible.
		io.k8s.display-name=Ruby 2.3
		io.openshift.expose-services=8080:http
		io.openshift.s2i.scripts-url=image:///usr/libexec/s2i
		io.openshift.tags=builder,ruby,ruby23,rh-ruby23
		io.s2i.scripts-url=image:///usr/libexec/s2i
		maintainer=SoftwareCollections.org <sclorg@redhat.com>
		name=rhscl/ruby-23-rhel7
		release=23
		summary=Platform for building and running Ruby 2.3 applications
		url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhscl/ruby-23-rhel7/images/2.3-23
		usage=s2i build https://github.com/sclorg/s2i-ruby-container.git --context-dir=2.4/test/puma-test-app/ rhscl/ruby-23-rhel7 ruby-sample-app
		vcs-ref=b6676600478a3dfa1e3352205d648b1c54697ecf
		vcs-type=git
		vendor=Red Hat, Inc.
		version=2.3
Environment:	PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
		container=oci
		SUMMARY=Platform for building and running Ruby 2.3 applications
		DESCRIPTION=Ruby 2.3 available as container is a base platform for building and running various Ruby 2.3 applications and frameworks. Ruby is the interpreted scripting language for quick and easy object-oriented programming. It has many features to process text files and to do system management tasks (as in Perl). It is simple, straight-forward, and extensible.
		STI_SCRIPTS_URL=image:///usr/libexec/s2i
		STI_SCRIPTS_PATH=/usr/libexec/s2i
		APP_ROOT=/opt/app-root
		HOME=/opt/app-root/src
		BASH_ENV=/opt/app-root/etc/scl_enable
		ENV=/opt/app-root/etc/scl_enable
		PROMPT_COMMAND=. /opt/app-root/etc/scl_enable
		NODEJS_SCL=rh-nodejs6
		RUBY_VERSION=2.3

root@ip-172-31-59-64: ~ # oc get is -o yaml -n openshift | grep message | grep "Internal error occurred" | wc -l
92
root@ip-172-31-59-64: ~ # oc get imagestreamtag | grep ruby
ruby:2.3                         registry.access.redhat.com/rhscl/ruby-23-rhel7@sha256:306e3cf5a6a9c9323c776fbe12ac35c6f621f44d4154ce0944a844bbec93e51f                     9 seconds ago    sha256:306e3cf5a6a9c9323c776fbe12ac35c6f621f44d4154ce0944a844bbec93e51f
ruby:2.4                         registry.access.redhat.com/rhscl/ruby-24-rhel7@sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d                     9 minutes ago    sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d
ruby:latest                      docker-registry.default.svc:5000/openshift/ruby@sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d                    9 minutes ago    sha256:da812edc2a0c0c8c28854daeb7629c733dcc5672c33d44f920ea8f1f2d6a058d

Comment 16 Ben Parees 2018-05-07 21:40:51 UTC
Ben, were there any known issues w/ the registry in the last week?  It seems like openshift has been getting 403ed while trying to access urls like:

http://access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/ose/3/containers/registry/openshift3/ruby-20-rhel7/manifests/1/latest

which i would expect to be open/not require authentication.


Hongkai is this being seen on other cluster installations performed by the QE team?

Comment 17 Hongkai Liu 2018-05-08 11:29:58 UTC
Let me check it today if we still have the problem.

Comment 18 Hongkai Liu 2018-05-08 12:43:03 UTC
Tested with
# yum list installed | grep openshift
atomic-openshift.x86_64       3.10.0-0.32.0.git.0.2b17fd0.el7

Much much better now.

# oc get is -o yaml -n openshift | grep message | grep "Internal error occurred" | wc -l
2


I guess those 2 are expected:
====================

  status:
    dockerImageRepository: docker-registry.default.svc:5000/openshift/jboss-eap71-openshift
    tags:
    - conditions:
      - generation: 2
        lastTransitionTime: 2018-05-08T11:57:45Z
        message: 'Internal error occurred: unknown: Not Found'
        reason: InternalError
        status: "False"
        type: ImportSuccess
      items: null
      tag: 1.0-TP


    dockerImageRepository: docker-registry.default.svc:5000/openshift/jenkins
    tags:
    - items:
      - created: 2018-05-08T11:57:19Z
        dockerImageReference: registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:a75802c902d7056e19cc4de6210e5312db396e638cebb8291058a4378199450a
        generation: 2
        image: sha256:a75802c902d7056e19cc4de6210e5312db396e638cebb8291058a4378199450a
      tag: "1"
    - conditions:
      - generation: 2
        lastTransitionTime: 2018-05-08T11:57:19Z
        message: 'Internal error occurred: error parsing HTTP 404 response body: invalid
          character ''F'' looking for beginning of value: "File not found.\""'
        reason: InternalError
        status: "False"
        type: ImportSuccess
      items: null
      tag: "2"

Comment 19 Ben Parees 2018-05-08 14:02:47 UTC
can you supply the full imagestream yaml? I want to look at those two failures slightly closer.

Comment 21 Ben Parees 2018-05-08 15:22:26 UTC
yes, those are both expected.

i'm going to move this to ON_QA in hopes that the failures were caused by registry hiccups but please reopen it if you experience this again.

Comment 22 Hongkai Liu 2018-05-08 17:53:33 UTC
Thanks, Ben.


Note You need to log in before you can comment on or make changes to this bug.