Bug 1480442

Summary: registry-console points to wrong image tag
Product: OpenShift Container Platform Reporter: Marko Myllynen <myllynen>
Component: InstallerAssignee: Scott Dodson <sdodson>
Status: CLOSED ERRATA QA Contact: Johnny Liu <jialiu>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.6.0CC: aos-bugs, bleanhar, bparees, jokerman, mfojtik, mmccomas, myllynen
Target Milestone: ---Keywords: NeedsTestCase, Reopened, Unconfirmed
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-ansible-3.7.0-0.126.4 Doc Type: Bug Fix
Doc Text:
The 'registry-console' image stream did not have a source tag specified and as such if the latest tag did not exist it would not properly import. The source tag has been added to the image stream ensuring that it imports properly.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 22:07:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marko Myllynen 2017-08-11 06:26:06 UTC
Description of problem:
In https://bugzilla.redhat.com/show_bug.cgi?id=1443588 it was stated:

latest - as well as no tag (which defaults to latest) :
**Users and scripts should not use the latest tag when dealing with openshift images**

However, when installing OCP 3.6 (3.6.173.0.5-3.git.0.522a92a.el7 / v3.6.173.0.5) and navigating to the Registry Console -> Images -> default/registry-console we see this error in disconnected environments:

dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found. Timestamp: 2017-08-10T15:51:43Z Error count: 2

So if the statement in the above BZ still holds it looks like this should be fixed (since, as explained in the BZ and elsewhere, latest can be for example OCP 3.4 or OCP 3.5 image, depending on the day).

Thanks.

Comment 1 Johnny Liu 2017-08-11 08:30:48 UTC
According to https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_hosted_templates/files/v3.6/enterprise/registry-console.yaml#L108, how did you let installer to pull "latest" image? Did you specifify openshift_cockpit_deployer_version=latest on purpose in your inventory host file?


In my env, it is working well.
# oc get po
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-p2pdw    1/1       Running   0          56m
registry-console-1-tfg3l   1/1       Running   0          55m
router-1-8mv6q             1/1       Running   0          57m


# oc describe po registry-console-1-tfg3l|grep Image
    Image:		registry.xxx.com/openshift3/registry-console:v3.6

Comment 2 Marko Myllynen 2017-08-11 09:02:07 UTC
Sorry for being unclear with the description.

The registry-console pod itself is running all ok, it is v3.6 / 5fcc300dfad7.

The issue is seen at the actual registry-console Web UI where in case of disconnected installation the tag "latest" hasn't been synced as per the above guidelines. Instead of syncing also "latest" (which can be an OCP 3.4 image) the Web UI should show v3.6.

Of course, if the policy is changing so that "latest" now should be used (against the previous statement) then there is no issue but currently documentation says that "latest" shouldn't be used.

Thanks.

Comment 3 Johnny Liu 2017-08-11 09:48:17 UTC
Could you run the following command:
# oc get is
NAME               DOCKER REPO                                                 TAGS      UPDATED
registry-console   docker-registry.default.svc:5000/default/registry-console   v3.6      



In my env, it is pointing to v3.6, but not latest.

Comment 4 Johnny Liu 2017-08-11 09:49:56 UTC
together with the output of `oc get is egistry-console -o yaml`

Comment 5 Marko Myllynen 2017-08-11 10:49:52 UTC
[root@master01 ~]# oc get is
NAME               DOCKER REPO                                                 TAGS      UPDATED
registry-console   docker-registry.default.svc:5000/default/registry-console   v3.6      
[root@master01 ~]# oc get is registry-console -o yaml
apiVersion: v1
kind: ImageStream
metadata:
  annotations:
    description: Atomic Registry console
    openshift.io/generated-by: OpenShiftNewApp
    openshift.io/image.dockerRepositoryCheck: 2017-08-11T10:44:46Z
  creationTimestamp: 2017-08-11T10:44:46Z
  generation: 2
  labels:
    app: registry-console
    createdBy: registry-console-template
  name: registry-console
  namespace: default
  resourceVersion: "2253"
  selfLink: /oapi/v1/namespaces/default/imagestreams/registry-console
  uid: 17536b78-7e82-11e7-9e3c-525400c0c40d
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations: null
    from:
      kind: DockerImage
      name: registry.example.com:5000/openshift3/registry-console
    generation: 2
    importPolicy: {}
    name: v3.6
    referencePolicy:
      type: Source
status:
  dockerImageRepository: docker-registry.default.svc:5000/default/registry-console
  tags:
  - conditions:
    - generation: 2
      lastTransitionTime: 2017-08-11T10:44:46Z
      message: dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest"
        not found
      reason: NotFound
      status: "False"
      type: ImportSuccess
    items: null
    tag: v3.6
[root@master01 ~]# 

Thanks,

Comment 6 Scott Dodson 2017-08-14 15:26:22 UTC
Which tags do exist for registry.example.com:5000/openshift3/registry-console?

skopeo inspect docker://registry.example.com:5000/openshift3/registry-console

I suspect that this is happening only because the v3.6 tag doesn't exist in your registry. If that's so can you please ensure that v3.6 tag exists and reproduce?

Comment 7 Marko Myllynen 2017-08-14 15:30:10 UTC
v3.6 is present:

registry.example.com:5000/openshift3/registry-console            v3.6
registry.example.com:5000/openshift3/registry-console            v3.6.173.0.5
registry.example.com:5000/openshift3/registry-console            v3.6.173.0.5-4

Comment 8 Scott Dodson 2017-08-14 16:00:46 UTC
It was present when this image stream was imported? Can you try `oc import registry-console` now?

Comment 9 Marko Myllynen 2017-08-14 16:12:11 UTC
(In reply to Scott Dodson from comment #8)
> It was present when this image stream was imported?

No, the local registry was repopulated just recently, only these tags have ever been present after that, and the latest installation after that operation is still using latest.

> Can you try `oc import registry-console` now?

Hmm?

[root@master01 ~]# oc whoami
system:admin
[root@master01 ~]# oc import registry-console
error: unknown command "registry-console"
See 'oc import -h' for help and examples.
[root@master01 ~]# 

Thanks.

Comment 10 Scott Dodson 2017-08-21 19:24:43 UTC
Sorry, I meant `oc import-image registry-console`

Comment 11 Marko Myllynen 2017-08-22 08:05:23 UTC
[root@master01 ~]# oc whoami
system:admin
[root@master01 ~]# oc import-image registry-console
error: the tag "latest" does not exist on the image stream - choose an existing tag to import or use the 'tag' command to create a new tag
[root@master01 ~]# 

https://registry-console-default.apps.test.example.com/registry#/images/default/registry-console still shows:

Image Stream
dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found. Timestamp: 2017-08-22T07:57:30Z Error count: 2 Edit image stream

Thanks.

Comment 12 Scott Dodson 2017-08-22 20:43:58 UTC
Ben,

Do image streams always implicitly attempt to import latest from their source when the specified tags are not found? As far as I can tell the v3.6 image stream tag is the only tag in the image stream definition however when the image stream was imported that tag did not exist and has been subsequently created however even now it's attempting to import the latest tag.

comment 5 is where i'm seeing signs of that.

Comment 13 Ben Parees 2017-08-22 20:48:06 UTC
I wouldn't have thought so, but we're going to have to ask Michal for now until I learn who knows this stuff :)

Comment 14 Scott Dodson 2017-08-22 21:22:31 UTC
Given neither QE nor I can reproduce this problem I'm going to put the onus on the reporter to reproduce after having verified that the 'v3.6' tag exists in the desired registry prior to installation. I think we've just gotten ourselves into an errant state that's not possible in an environment that has the expected tags.

Maybe Michal decides there's a bug to chase here but I'm pretty sure it's not an installer bug.

Comment 15 Michal Fojtik 2017-08-23 08:17:24 UTC
(In reply to Scott Dodson from comment #12)
> Ben,
> 
> Do image streams always implicitly attempt to import latest from their
> source when the specified tags are not found? As far as I can tell the v3.6
> image stream tag is the only tag in the image stream definition however when
> the image stream was imported that tag did not exist and has been
> subsequently created however even now it's attempting to import the latest
> tag.
> 
> comment 5 is where i'm seeing signs of that.

If the tag is not given, the import-image will default to `:latest`. If you run `oc import-image registry-console:v3.6` is should succeed?

Comment 16 Marko Myllynen 2017-08-23 12:31:52 UTC
To recap, this is a disconnected installation with not access to redhat.com from any node, the local registry.example.com has these images:

# docker images | grep registry-console       
registry.access.redhat.com/openshift3/registry-console           v3.6                00d95b5a257e        11 days ago         226.2 MB
registry.access.redhat.com/openshift3/registry-console           v3.6.173.0.5        00d95b5a257e        11 days ago         226.2 MB
registry.access.redhat.com/openshift3/registry-console           v3.6.173.0.5-6      00d95b5a257e        11 days ago         226.2 MB

And after the installation on the first master we see:

[root@master01 ~]# oc import-image registry-console:v3.6
The import completed with errors.

Name:			registry-console
Namespace:		default
Created:		2 hours ago
Labels:			app=registry-console
			createdBy=registry-console-template
Description:		Atomic Registry console
Annotations:		openshift.io/generated-by=OpenShiftNewApp
			openshift.io/image.dockerRepositoryCheck=2017-08-23T10:22:34Z
Docker Pull Spec:	docker-registry.default.svc:5000/default/registry-console
Image Lookup:		local=false
Unique Images:		0
Tags:			1

v3.6
  tagged from registry.example.com:5000/openshift3/registry-console

  ! error: Import failed (NotFound): dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found
      2 hours ago

error: tag latest failed: dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found
[root@master01 ~]# 

This is 100% reproducible, please let me know if you need more details.

Thanks.

Comment 17 Brenton Leanhardt 2017-08-31 15:46:16 UTC
I'm passing this to Michal's team based on Comment #15 and Comment #16.

Comment 19 Ben Parees 2017-09-11 14:15:28 UTC
As michal said in comment 15:
If the tag is not given, the import-image will default to `:latest`. If you run `oc import-image registry-console:v3.6` is should succeed?

So here's the sequence of events and the fix:

1) imagestream was created with a dockerImageRepository of "docker-registry.default.svc:5000/default/registry-console"
2) imagestream also contained an explicit tag to be imported: registry.example.com:5000/openshift3/registry-console:v3.6
3) at the time the IS was created, the v3.6 tag didn't exist, so it wasn't imported.  Other tags have have been imported, from the registry.default.svc location (per the dockerImageRepository value for the IS)
4) the v3.6 tag came into existence in the registry

at this point, the correct action to take would have been what Michal said, explicitly requesting an oc import-image for the specific tag (v3.6).

I think "oc import-image registry-console --all" might have worked also.

So I'm closing this as working as designed.

Comment 20 Marko Myllynen 2017-09-12 08:40:47 UTC
(In reply to Ben Parees from comment #19)
> As michal said in comment 15:
> If the tag is not given, the import-image will default to `:latest`. If you
> run `oc import-image registry-console:v3.6` is should succeed?

No, this fails as well:

[root@master01 ~]# oc import-image registry-console:v3.6
The import completed with errors.

Name:			registry-console
Namespace:		default
Created:		6 minutes ago
Labels:			app=registry-console
			createdBy=registry-console-template
Description:		Atomic Registry console
Annotations:		openshift.io/generated-by=OpenShiftNewApp
			openshift.io/image.dockerRepositoryCheck=2017-09-12T08:29:26Z
Docker Pull Spec:	docker-registry.default.svc:5000/default/registry-console
Image Lookup:		local=false
Unique Images:		0
Tags:			1

v3.6
  tagged from registry.example.com:5000/openshift3/registry-console

  ! error: Import failed (NotFound): dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found
      6 minutes ago

error: tag latest failed: dockerimage.image.openshift.io "registry.example.com:5000/openshift3/registry-console:latest" not found

So why look for latest if v3.6 was requested?

> So here's the sequence of events and the fix:
> 
> So I'm closing this as working as designed.

No, the point of this BZ was to make this work out of the box after the initial installation, broken-after-install surely can't be as designed. If additional steps are needed to unbreak things then the installer should do them and please spell out what they would be as the previous suggestions haven't helped.

Thanks.

Comment 21 Ben Parees 2017-09-12 19:11:46 UTC
Since you're running in disconnected mode, what was responsible for populating the registry you are using?  I would like to understand why it does not have a latest tag.  Creating that tag would be a logical workaround to your current problem.

(However the import-image behavior itself is a separate bug, it should only import the tag you've requested.  I've reported that issue here: https://github.com/openshift/origin/issues/16310).

Comment 22 Marko Myllynen 2017-09-12 19:18:52 UTC
(In reply to Ben Parees from comment #21)
> Since you're running in disconnected mode, what was responsible for
> populating the registry you are using?  I would like to understand why it
> does not have a latest tag.  Creating that tag would be a logical workaround
> to your current problem.

The latest tag is deliberately not synced to the local registry by local tooling for the reason stated in comment 0:

"
In https://bugzilla.redhat.com/show_bug.cgi?id=1443588 it was stated:

latest - as well as no tag (which defaults to latest) :
**Users and scripts should not use the latest tag when dealing with openshift images**
"

As we know, latest might refer to a 3.6 version of an image this week, a 3.4 version next week, and a 3.5 version the week after that. This is the crux of the issue, I don't see syncing latest as a reliable workaround, especially since it goes against the recommendation stated earlier.

Thanks.

Comment 23 Ben Parees 2017-09-12 19:36:17 UTC
Using it and having it exist in your registry so that the import-image command does not fail are two different things.

Comment 24 Marko Myllynen 2017-09-12 19:44:59 UTC
(In reply to Ben Parees from comment #23)
> Using it and having it exist in your registry so that the import-image
> command does not fail are two different things.

Our official documentation states:

Do not use the latest tag for any official OpenShift Container Platform images.

I can't see it helping our customers or users that we'd start adding exceptions here for the sake of an individual subcommand, there's been enough confusion around the latest tag already.

Thanks.

Comment 25 Ben Parees 2017-09-12 19:51:26 UTC
I'm suggesting a workaround to you.  I already acknowledged the import-image command's behavior is a bug and opened an issue on it.

Beyond that, it's an installer problem and I've transferred the bug accordingly.  (the registry-console imagestream is not defined correctly, it should not have a dockerImageRepository reference at all and it should only contain explicit spec.tags references to the image version it intends to consume).

Comment 26 Marko Myllynen 2017-09-12 19:54:20 UTC
Thanks for digging out all the details and filing the upstream issue!

Comment 27 Scott Dodson 2017-09-12 21:20:23 UTC
Ben,

Thanks for the analysis. The template is created by running oc newapp against a template with the following image stream definition.

https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_hosted_templates/files/v3.6/enterprise/registry-console.yaml#L81-L93

Is there a change that should be made to that definition or is there something wrong with `oc newapp`?

--
Scott

Comment 29 Ben Parees 2017-09-12 23:24:11 UTC
Scott, it seems like that definition: https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_hosted_templates/files/v3.6/enterprise/registry-console.yaml#L92  should have a version tag in it, otherwise it's just going to pick up :latest which i assume is not desired.  (Since the bug report does show the imagestreamtag pointing to :v3.6, perhaps there is some other logic that filled in the tag?)

And in hindsight I think that is the core issue here, not what I stated earlier.

The tag in question from comment 5 is:

spec:
  lookupPolicy:
    local: false
  tags:
  - annotations: null
    from:
      kind: DockerImage
      name: registry.example.com:5000/openshift3/registry-console
    generation: 2
    importPolicy: {}
    name: v3.6
    referencePolicy:
      type: Source

Note that although the imagestreamtag is named "v3.6", the "from" section does not specify a docker image tag, which means it's going to default to latest.

That is why the import fails.

it should be:
      name: registry.example.com:5000/openshift3/registry-console:v3.6

(assuming that is the correct name of the dockerimage/tag that exists in the registry.example.com registry)

Sorry I missed this earlier.

Comment 34 errata-xmlrpc 2017-11-28 22:07:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188