Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1807730 - Registry server providing invalid information for aodh-notifier image
Summary: Registry server providing invalid information for aodh-notifier image
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-containers
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Jon Schlueter
QA Contact: Arik Chernetsky
URL:
Whiteboard:
: 1807612 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-27 04:42 UTC by Lars Kellogg-Stedman
Modified: 2020-07-11 22:03 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 4894021 0 None None None 2020-03-10 17:45:37 UTC

Description Lars Kellogg-Stedman 2020-02-27 04:42:12 UTC
Description of problem:

Running:

  openstack overcloud container image prepare \
    --namespace=registry.access.redhat.com/rhosp13 \
    --push-destination=192.168.24.1:8787 \
    --prefix=openstack- \
    --tag-from-label {version}-{release} \
    --output-env-file=env.yaml \
    --output-images-file=images.yaml

Is failing with:


  Image registry.access.redhat.com/rhosp13/openstack-aodh-notifier has no tag 13.0-100.1582098610.
  Available tags: 13.0-85, 13.0-86, 13.0-83, 13.0-100, 13.0, 13.0-56, 13.0-55, 13.0-76, 13.0-72, 13.0-39, 13.0-96, 13.0-100.1580118203, 13.0-62.1543534128, 13.0-68, 13.0-72.1557945115, 13.0-46, 13.0-62, 13.0-60, 13.0-61, 13.0-66, 13.0-64, 13.0-68.1554788878, latest


And indeed, running:

  skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest

Shows that while the "release" label is "100.1582098610", there is no matching tag in RepoTags.

There is some indication this may only be affecting *some* of the registry servers; conversations on internal irc suggest that some folks were hitting this and some were not.

Comment 2 Brendan Shephard 2020-02-27 05:51:39 UTC
The traceback:

~~~
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 439, in take_action
    template_file=parsed_args.template_file
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py", line 80, in container_images_prepare
    images, tag_from_label)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 344, in discover_image_tags
    discover_args):
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
ImageUploaderException: Image registry.access.redhat.com/rhosp13/openstack-aodh-notifier has no tag 13.0-100.1582098610.
Available tags: 13.0-85, 13.0-86, 13.0-83, 13.0-100, 13.0, 13.0-56, 13.0-55, 13.0-76, 13.0-72, 13.0-39, 13.0-96, 13.0-100.1580118203, 13.0-62.1543534128, 13.0-68, 13.0-72.1557945115, 13.0-46, 13.0-62, 13.0-60, 13.0-61, 13.0-66, 13.0-64, 
13.0-68.1554788878, latest
~~~


Happens when you include --tag-from-label {version}-{release}

Without it, it works fine and just pulls the latest containers.

Comment 3 Brendan Shephard 2020-02-27 06:21:08 UTC
I was seeing that container for a while there:

~~~
$ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags                                                                                       [140/1988]
[                                                                                                                     
  "13.0-85",                                                                                                          
  "13.0-86",                                                                                                          
  "13.0-83",                                                                                                          
  "13.0-100",                                                                                                         
  "13.0",                                                                                                             
  "13.0-56",                                                                                                          
  "13.0-55",                                                                                                          
  "13.0-76",                                                                                                          
  "13.0-72",                                                                                                          
  "13.0-39",                                                                                                          
  "13.0-100.1582098610",  <<<<--- There is is                                                                                                                                                                                                                   
  "13.0-96",                                                                                                          
  "13.0-100.1580118203",                                                                                              
  "13.0-62.1543534128",                                                                                               
  "13.0-68",                                                                                                          
  "13.0-72.1557945115",                                                                                               
  "13.0-46",                                                                                                          
  "13.0-62",                                                                                                          
  "13.0-60",                                                                                                          
  "13.0-61",                                                                                                          
  "13.0-66",                                                                                                          
  "13.0-64",                                                                                                          
  "13.0-68.1554788878",                                                                                               
  "latest"                                                                                                            
] 
~~~



But now:
~~~
$ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags
[
  "13.0-85",
  "13.0-86",
  "13.0-83",
  "13.0-100",
  "13.0",
  "13.0-56",
  "13.0-55",
  "13.0-76",
  "13.0-72",
  "13.0-39",
  "13.0-96",
  "13.0-100.1580118203",
  "13.0-62.1543534128",
  "13.0-68",
  "13.0-72.1557945115",
  "13.0-46",
  "13.0-62",
  "13.0-60",
  "13.0-61",
  "13.0-66",
  "13.0-64",
  "13.0-68.1554788878",
  "latest"
]
~~~

Something with the registry maybe?

Comment 4 Jon Schlueter 2020-02-27 08:06:29 UTC
possible issue with registry searching 

I did verify that we can pull the image 

podman pull registry.access.redhat.com/rhosp13/openstack-aodh-notifier:13.0-100.1582098610

but that does not help installation issues directly

Comment 6 Alfredo Moralejo 2020-02-28 17:27:55 UTC
I've just ran following script for rhosp13 and all containers looks sane right now:

podman search --limit 1000 "registry.redhat.io/rhosp13" | awk '{ print $2 }' | sed "s/registry.redhat.io\///g"|while read a
do
  RELEASE=$(skopeo inspect docker://registry.access.redhat.com/${a}:latest| jq .Labels.release|tr -d '"')
  echo " $a:$RELEASE"
  skopeo inspect docker://registry.access.redhat.com/${a}:latest| jq .RepoTags|grep -q $RELEASE
  if [ $? -ne 0 ]; then
  echo "Failed $a:$RELEASE"
fi
done

It seems all container are now correct, at least from my test.

Comment 7 Alfredo Moralejo 2020-02-28 17:43:36 UTC
*** Bug 1807612 has been marked as a duplicate of this bug. ***

Comment 9 David Hill 2020-03-10 17:44:38 UTC
openstack overcloud container image prepare --namespace=registry.access.redhat.com/rhosp13 --push-destination={--PUSH LOCATION--}:8787 --prefix=openstack- --tag-from-label {version}-{release} --output-env-file=/home/stack/templates/overcloud_images-{--DATE--}.yaml --output-images-file /home/stack/local_registry_images-{--DATE--}.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml --set ceph_namespace=registry.access.redhat.com/rhceph --set ceph_image=rhceph-3-rhel7 --tag-from-label {version}-{release} -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovn-dvr-ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/barbican.yaml

i receive the following error message; 
Image registry.access.redhat.com/rhosp13/openstack-aodh-evaluator has no tag 13.0-107.

Comment 10 David Hill 2020-03-10 17:52:48 UTC
I just tried the same command as the customer and I didn't hit the issue:

~~~
[stack@undercloud-0-rhosp13 ~]$ openstack overcloud container image prepare --namespace=registry.access.redhat.com/rhosp13 --push-destination=192.0.2.1:8787 --prefix=openstack- --tag-from-label {version}-{release} --output-env-file=/home/stack/overcloud_images.yaml --output-images-file /home/stack/local_registry_images-{--DATE--}.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml --set ceph_namespace=registry.access.redhat.com/rhceph --set ceph_image=rhceph-3-rhel7 --tag-from-label {version}-{release} -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovn-dvr-ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/barbican.yaml
container_images:
- imagename: registry.access.redhat.com/rhosp13/openstack-aodh-api:13.0-107
  push_destination: 192.0.2.1:8787
- imagename: registry.access.redhat.com/rhosp13/openstack-aodh-evaluator:13.0-107
  push_destination: 192.0.2.1:8787
[...]
~~~

My current customer is in EMEA / UK ... Is it possible that their mirrors / CDNs / etc are catching up / broken ?

Comment 11 David Hill 2020-03-11 12:36:55 UTC
Customer updated the case with this:
~~~
I have re-run the command this morning and it has worked as expected. When we looked at the container registry via https://catalog.redhat.com/software/containers/explore we noticed that the containers we were having issues with had only been updated a couple of hours prior, so maybe the sync between registry.redhat.com and registry.redhat.io takes a while.
~~~

Comment 13 Jon Schlueter 2020-03-12 15:05:04 UTC
Possible idea for tripleo upstream to make it more bullet proof

working with the assumption that it is a data race with the registry, that the :<tag> fetched form the registry is not on the RepoTags list that skopeo inspect gives

so the situation I observed the first time it was reported was that the image available via inspection had a <version>-<release> tag that was fetchable but not in the tags list

$ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq ".Labels.release"
"100.1582098610"

$ curl -L https://registry.access.redhat.com/v2/rhosp13/openstack-aodh-notifier/tags/list
{"name": "rhosp13/openstack-aodh-notifier", "tags": ["13.0-85", "13.0-86", "13.0-83", "13.0-100", "13.0", "13.0-56", "13.0-55", "13.0-76", "13.0-72", "13.0-39", "13.0-96", "13.0-100.1580118203", "13.0-62.1543534128", "13.0-68", "13.0-72.1557945115", "13.0-46", "13.0-62", "13.0-60", "13.0-61", "13.0-66", "13.0-64", "13.0-68.1554788878", "latest"]}

So the idea would be to implement a fallback instead of failing hard of inspecting the desired image directly

so I think it was trying to use RepoTags data from skopeo inspect or something like it to check to see if image was available and failing because it was not in the list.  Fallback would be to then attempt to skopeo inspect the explicit image with tag we want to see if it was available and valid, instead of just failing because of inconsistency.

Comment 14 David Hill 2020-03-19 13:09:36 UTC
skopeo inspect docker://registry.redhat.io/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags
[
  "13.0-85",
  "13.0-86",
  "13.0-83",
  "13.0-100",
  "13.0-109",
  "13.0",
  "13.0-56",
  "13.0-55",
  "13.0-76",
  "13.0-72",
  "13.0-39",
  "13.0-100.1582098610",
  "13.0-96",
  "13.0-100.1580118203",
  "13.0-62.1543534128",
  "13.0-68",
  "13.0-72.1557945115",
  "13.0-46",
  "13.0-62",
  "13.0-60",
  "13.0-61",
  "13.0-66",
  "13.0-64",
  "13.0-68.1554788878",
  "latest"
]



[root@nps ~]# skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags
[
  "13.0-85",
  "13.0-86",
  "13.0-83",
  "13.0-100",
  "13.0-109",
  "13.0",
  "13.0-56",
  "13.0-55",
  "13.0-76",
  "13.0-72",
  "13.0-39",
  "13.0-100.1582098610",
  "13.0-96",
  "13.0-100.1580118203",
  "13.0-62.1543534128",
  "13.0-68",
  "13.0-72.1557945115",
  "13.0-46",
  "13.0-62",
  "13.0-60",
  "13.0-61",
  "13.0-66",
  "13.0-64",
  "13.0-68.1554788878",
  "latest"
]
[root@nps ~]# skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq ".Labels.release"
"109"

[root@nps ~]# skopeo inspect docker://registry.redhat.io/rhosp13/openstack-aodh-notifier:latest | jq ".Labels.release"
"109"

Comment 15 David Hill 2020-03-19 13:10:25 UTC
source /home/stack/stackrc && openstack overcloud container image prepare --namespace=10.XX.XX.XX:8787/rhosp13 --push-destination="10.XX.XX.XX":8787 --prefix=openstack- --tag-from-label {version}-{release} -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/neutron-sriov.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/neutron-ovs-dpdk.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/cinder-backup.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/collectd.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/fluentd.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sensu-client.yaml -r  /home/stack/templates/roles_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml --set ceph_namespace=10.XX.XX.XX:8787/rhceph --set ceph_image=rhceph-3-rhel7 --tag-from-label {version}-{release} --output-env-file=/home/stack/templates/env_files/overcloud_images.yaml --output-images-file /home/stack/local_registry_images.yaml

Error:
Image 10.XX.XX.XX:8787/rhosp13/openstack-aodh-notifier has no tag 13.0-100.1582098610.

Comment 16 Jon Schlueter 2020-05-06 14:00:26 UTC
issue was identified as possible publishing issue, and has since been resolved

$ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq ".Labels.release"
"115"

$ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags | grep 115
  "13.0-115",
  "13.0-72.1557945115",


Additional verification by impacted parties would be good just need to run the above skopeo inspect commands from your system

Comment 17 Jon Schlueter 2020-05-06 14:04:11 UTC
kholtz@redhat.com
dhill@redhat.com
afariasa@redhat.com

Can you verify that this is resolved now?

Comment 18 Miguel Garcia 2020-06-25 14:58:58 UTC
Tested this after yesterday's release of new containers:

$ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq ".Labels.release"
"120"

$ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags | grep 120
  "13.0-120",

Everything seems to be in order. Marking VERIFIED.


Note You need to log in before you can comment on or make changes to this bug.