Description of problem: In https://bugzilla.redhat.com/show_bug.cgi?id=1443588 it was stated: latest - as well as no tag (which defaults to latest) : **Users and scripts should not use the latest tag when dealing with openshift images** However, when installing OCP 3.6 (3.6.173.0.5-3.git.0.522a92a.el7 / v3.6.173.0.5) and navigating to the Registry Console -> Images -> default/registry-console we see this error in disconnected environments: dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found. Timestamp: 2017-08-10T15:51:43Z Error count: 2 So if the statement in the above BZ still holds it looks like this should be fixed (since, as explained in the BZ and elsewhere, latest can be for example OCP 3.4 or OCP 3.5 image, depending on the day). Thanks.
According to https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_hosted_templates/files/v3.6/enterprise/registry-console.yaml#L108, how did you let installer to pull "latest" image? Did you specifify openshift_cockpit_deployer_version=latest on purpose in your inventory host file? In my env, it is working well. # oc get po NAME READY STATUS RESTARTS AGE docker-registry-1-p2pdw 1/1 Running 0 56m registry-console-1-tfg3l 1/1 Running 0 55m router-1-8mv6q 1/1 Running 0 57m # oc describe po registry-console-1-tfg3l|grep Image Image: registry.xxx.com/openshift3/registry-console:v3.6
Sorry for being unclear with the description. The registry-console pod itself is running all ok, it is v3.6 / 5fcc300dfad7. The issue is seen at the actual registry-console Web UI where in case of disconnected installation the tag "latest" hasn't been synced as per the above guidelines. Instead of syncing also "latest" (which can be an OCP 3.4 image) the Web UI should show v3.6. Of course, if the policy is changing so that "latest" now should be used (against the previous statement) then there is no issue but currently documentation says that "latest" shouldn't be used. Thanks.
Could you run the following command: # oc get is NAME DOCKER REPO TAGS UPDATED registry-console docker-registry.default.svc:5000/default/registry-console v3.6 In my env, it is pointing to v3.6, but not latest.
together with the output of `oc get is egistry-console -o yaml`
[root@master01 ~]# oc get is NAME DOCKER REPO TAGS UPDATED registry-console docker-registry.default.svc:5000/default/registry-console v3.6 [root@master01 ~]# oc get is registry-console -o yaml apiVersion: v1 kind: ImageStream metadata: annotations: description: Atomic Registry console openshift.io/generated-by: OpenShiftNewApp openshift.io/image.dockerRepositoryCheck: 2017-08-11T10:44:46Z creationTimestamp: 2017-08-11T10:44:46Z generation: 2 labels: app: registry-console createdBy: registry-console-template name: registry-console namespace: default resourceVersion: "2253" selfLink: /oapi/v1/namespaces/default/imagestreams/registry-console uid: 17536b78-7e82-11e7-9e3c-525400c0c40d spec: lookupPolicy: local: false tags: - annotations: null from: kind: DockerImage name: registry.example.com:5000/openshift3/registry-console generation: 2 importPolicy: {} name: v3.6 referencePolicy: type: Source status: dockerImageRepository: docker-registry.default.svc:5000/default/registry-console tags: - conditions: - generation: 2 lastTransitionTime: 2017-08-11T10:44:46Z message: dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found reason: NotFound status: "False" type: ImportSuccess items: null tag: v3.6 [root@master01 ~]# Thanks,
Which tags do exist for registry.example.com:5000/openshift3/registry-console? skopeo inspect docker://registry.example.com:5000/openshift3/registry-console I suspect that this is happening only because the v3.6 tag doesn't exist in your registry. If that's so can you please ensure that v3.6 tag exists and reproduce?
v3.6 is present: registry.example.com:5000/openshift3/registry-console v3.6 registry.example.com:5000/openshift3/registry-console v3.6.173.0.5 registry.example.com:5000/openshift3/registry-console v3.6.173.0.5-4
It was present when this image stream was imported? Can you try `oc import registry-console` now?
(In reply to Scott Dodson from comment #8) > It was present when this image stream was imported? No, the local registry was repopulated just recently, only these tags have ever been present after that, and the latest installation after that operation is still using latest. > Can you try `oc import registry-console` now? Hmm? [root@master01 ~]# oc whoami system:admin [root@master01 ~]# oc import registry-console error: unknown command "registry-console" See 'oc import -h' for help and examples. [root@master01 ~]# Thanks.
Sorry, I meant `oc import-image registry-console`
[root@master01 ~]# oc whoami system:admin [root@master01 ~]# oc import-image registry-console error: the tag "latest" does not exist on the image stream - choose an existing tag to import or use the 'tag' command to create a new tag [root@master01 ~]# https://registry-console-default.apps.test.example.com/registry#/images/default/registry-console still shows: Image Stream dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found. Timestamp: 2017-08-22T07:57:30Z Error count: 2 Edit image stream Thanks.
Ben, Do image streams always implicitly attempt to import latest from their source when the specified tags are not found? As far as I can tell the v3.6 image stream tag is the only tag in the image stream definition however when the image stream was imported that tag did not exist and has been subsequently created however even now it's attempting to import the latest tag. comment 5 is where i'm seeing signs of that.
I wouldn't have thought so, but we're going to have to ask Michal for now until I learn who knows this stuff :)
Given neither QE nor I can reproduce this problem I'm going to put the onus on the reporter to reproduce after having verified that the 'v3.6' tag exists in the desired registry prior to installation. I think we've just gotten ourselves into an errant state that's not possible in an environment that has the expected tags. Maybe Michal decides there's a bug to chase here but I'm pretty sure it's not an installer bug.
(In reply to Scott Dodson from comment #12) > Ben, > > Do image streams always implicitly attempt to import latest from their > source when the specified tags are not found? As far as I can tell the v3.6 > image stream tag is the only tag in the image stream definition however when > the image stream was imported that tag did not exist and has been > subsequently created however even now it's attempting to import the latest > tag. > > comment 5 is where i'm seeing signs of that. If the tag is not given, the import-image will default to `:latest`. If you run `oc import-image registry-console:v3.6` is should succeed?
To recap, this is a disconnected installation with not access to redhat.com from any node, the local registry.example.com has these images: # docker images | grep registry-console registry.access.redhat.com/openshift3/registry-console v3.6 00d95b5a257e 11 days ago 226.2 MB registry.access.redhat.com/openshift3/registry-console v3.6.173.0.5 00d95b5a257e 11 days ago 226.2 MB registry.access.redhat.com/openshift3/registry-console v3.6.173.0.5-6 00d95b5a257e 11 days ago 226.2 MB And after the installation on the first master we see: [root@master01 ~]# oc import-image registry-console:v3.6 The import completed with errors. Name: registry-console Namespace: default Created: 2 hours ago Labels: app=registry-console createdBy=registry-console-template Description: Atomic Registry console Annotations: openshift.io/generated-by=OpenShiftNewApp openshift.io/image.dockerRepositoryCheck=2017-08-23T10:22:34Z Docker Pull Spec: docker-registry.default.svc:5000/default/registry-console Image Lookup: local=false Unique Images: 0 Tags: 1 v3.6 tagged from registry.example.com:5000/openshift3/registry-console ! error: Import failed (NotFound): dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found 2 hours ago error: tag latest failed: dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found [root@master01 ~]# This is 100% reproducible, please let me know if you need more details. Thanks.
I'm passing this to Michal's team based on Comment #15 and Comment #16.
As michal said in comment 15: If the tag is not given, the import-image will default to `:latest`. If you run `oc import-image registry-console:v3.6` is should succeed? So here's the sequence of events and the fix: 1) imagestream was created with a dockerImageRepository of "docker-registry.default.svc:5000/default/registry-console" 2) imagestream also contained an explicit tag to be imported: registry.example.com:5000/openshift3/registry-console:v3.6 3) at the time the IS was created, the v3.6 tag didn't exist, so it wasn't imported. Other tags have have been imported, from the registry.default.svc location (per the dockerImageRepository value for the IS) 4) the v3.6 tag came into existence in the registry at this point, the correct action to take would have been what Michal said, explicitly requesting an oc import-image for the specific tag (v3.6). I think "oc import-image registry-console --all" might have worked also. So I'm closing this as working as designed.
(In reply to Ben Parees from comment #19) > As michal said in comment 15: > If the tag is not given, the import-image will default to `:latest`. If you > run `oc import-image registry-console:v3.6` is should succeed? No, this fails as well: [root@master01 ~]# oc import-image registry-console:v3.6 The import completed with errors. Name: registry-console Namespace: default Created: 6 minutes ago Labels: app=registry-console createdBy=registry-console-template Description: Atomic Registry console Annotations: openshift.io/generated-by=OpenShiftNewApp openshift.io/image.dockerRepositoryCheck=2017-09-12T08:29:26Z Docker Pull Spec: docker-registry.default.svc:5000/default/registry-console Image Lookup: local=false Unique Images: 0 Tags: 1 v3.6 tagged from registry.example.com:5000/openshift3/registry-console ! error: Import failed (NotFound): dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found 6 minutes ago error: tag latest failed: dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found So why look for latest if v3.6 was requested? > So here's the sequence of events and the fix: > > So I'm closing this as working as designed. No, the point of this BZ was to make this work out of the box after the initial installation, broken-after-install surely can't be as designed. If additional steps are needed to unbreak things then the installer should do them and please spell out what they would be as the previous suggestions haven't helped. Thanks.
Since you're running in disconnected mode, what was responsible for populating the registry you are using? I would like to understand why it does not have a latest tag. Creating that tag would be a logical workaround to your current problem. (However the import-image behavior itself is a separate bug, it should only import the tag you've requested. I've reported that issue here: https://github.com/openshift/origin/issues/16310).
(In reply to Ben Parees from comment #21) > Since you're running in disconnected mode, what was responsible for > populating the registry you are using? I would like to understand why it > does not have a latest tag. Creating that tag would be a logical workaround > to your current problem. The latest tag is deliberately not synced to the local registry by local tooling for the reason stated in comment 0: " In https://bugzilla.redhat.com/show_bug.cgi?id=1443588 it was stated: latest - as well as no tag (which defaults to latest) : **Users and scripts should not use the latest tag when dealing with openshift images** " As we know, latest might refer to a 3.6 version of an image this week, a 3.4 version next week, and a 3.5 version the week after that. This is the crux of the issue, I don't see syncing latest as a reliable workaround, especially since it goes against the recommendation stated earlier. Thanks.
Using it and having it exist in your registry so that the import-image command does not fail are two different things.
(In reply to Ben Parees from comment #23) > Using it and having it exist in your registry so that the import-image > command does not fail are two different things. Our official documentation states: Do not use the latest tag for any official OpenShift Container Platform images. I can't see it helping our customers or users that we'd start adding exceptions here for the sake of an individual subcommand, there's been enough confusion around the latest tag already. Thanks.
I'm suggesting a workaround to you. I already acknowledged the import-image command's behavior is a bug and opened an issue on it. Beyond that, it's an installer problem and I've transferred the bug accordingly. (the registry-console imagestream is not defined correctly, it should not have a dockerImageRepository reference at all and it should only contain explicit spec.tags references to the image version it intends to consume).
Thanks for digging out all the details and filing the upstream issue!
Ben, Thanks for the analysis. The template is created by running oc newapp against a template with the following image stream definition. https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_hosted_templates/files/v3.6/enterprise/registry-console.yaml#L81-L93 Is there a change that should be made to that definition or is there something wrong with `oc newapp`? -- Scott
Scott, it seems like that definition: https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_hosted_templates/files/v3.6/enterprise/registry-console.yaml#L92 should have a version tag in it, otherwise it's just going to pick up :latest which i assume is not desired. (Since the bug report does show the imagestreamtag pointing to :v3.6, perhaps there is some other logic that filled in the tag?) And in hindsight I think that is the core issue here, not what I stated earlier. The tag in question from comment 5 is: spec: lookupPolicy: local: false tags: - annotations: null from: kind: DockerImage name: registry.example.com:5000/openshift3/registry-console generation: 2 importPolicy: {} name: v3.6 referencePolicy: type: Source Note that although the imagestreamtag is named "v3.6", the "from" section does not specify a docker image tag, which means it's going to default to latest. That is why the import fails. it should be: name: registry.example.com:5000/openshift3/registry-console:v3.6 (assuming that is the correct name of the dockerimage/tag that exists in the registry.example.com registry) Sorry I missed this earlier.
https://github.com/openshift/openshift-ansible/pull/5395
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188