Bug 1807730
| Summary: | Registry server providing invalid information for aodh-notifier image | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Lars Kellogg-Stedman <lars> |
| Component: | openstack-containers | Assignee: | Jon Schlueter <jschluet> |
| Status: | CLOSED EOL | QA Contact: | Arik Chernetsky <achernet> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 13.0 (Queens) | CC: | afariasa, amoralej, bshephar, dhill, jjoyce, jschluet, kholtz, m.andre, mburns, mgarciac, satmakur, shrjoshi, ssigwald, tvignaud |
| Target Milestone: | --- | Keywords: | TestOnly, Triaged, ZStream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-07-10 17:19:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The traceback:
~~~
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
ret_val = super(OpenStackShell, self).run(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
result = self.run_subcommand(remainder)
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
ret_value = super(OpenStackShell, self).run_subcommand(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
return_code = self.take_action(parsed_args) or 0
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 439, in take_action
template_file=parsed_args.template_file
File "/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py", line 80, in container_images_prepare
images, tag_from_label)
File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 344, in discover_image_tags
discover_args):
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 250, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
raise self._value
ImageUploaderException: Image registry.access.redhat.com/rhosp13/openstack-aodh-notifier has no tag 13.0-100.1582098610.
Available tags: 13.0-85, 13.0-86, 13.0-83, 13.0-100, 13.0, 13.0-56, 13.0-55, 13.0-76, 13.0-72, 13.0-39, 13.0-96, 13.0-100.1580118203, 13.0-62.1543534128, 13.0-68, 13.0-72.1557945115, 13.0-46, 13.0-62, 13.0-60, 13.0-61, 13.0-66, 13.0-64,
13.0-68.1554788878, latest
~~~
Happens when you include --tag-from-label {version}-{release}
Without it, it works fine and just pulls the latest containers.
I was seeing that container for a while there: ~~~ $ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags [140/1988] [ "13.0-85", "13.0-86", "13.0-83", "13.0-100", "13.0", "13.0-56", "13.0-55", "13.0-76", "13.0-72", "13.0-39", "13.0-100.1582098610", <<<<--- There is is "13.0-96", "13.0-100.1580118203", "13.0-62.1543534128", "13.0-68", "13.0-72.1557945115", "13.0-46", "13.0-62", "13.0-60", "13.0-61", "13.0-66", "13.0-64", "13.0-68.1554788878", "latest" ] ~~~ But now: ~~~ $ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags [ "13.0-85", "13.0-86", "13.0-83", "13.0-100", "13.0", "13.0-56", "13.0-55", "13.0-76", "13.0-72", "13.0-39", "13.0-96", "13.0-100.1580118203", "13.0-62.1543534128", "13.0-68", "13.0-72.1557945115", "13.0-46", "13.0-62", "13.0-60", "13.0-61", "13.0-66", "13.0-64", "13.0-68.1554788878", "latest" ] ~~~ Something with the registry maybe? possible issue with registry searching I did verify that we can pull the image podman pull registry.access.redhat.com/rhosp13/openstack-aodh-notifier:13.0-100.1582098610 but that does not help installation issues directly I've just ran following script for rhosp13 and all containers looks sane right now:
podman search --limit 1000 "registry.redhat.io/rhosp13" | awk '{ print $2 }' | sed "s/registry.redhat.io\///g"|while read a
do
RELEASE=$(skopeo inspect docker://registry.access.redhat.com/${a}:latest| jq .Labels.release|tr -d '"')
echo " $a:$RELEASE"
skopeo inspect docker://registry.access.redhat.com/${a}:latest| jq .RepoTags|grep -q $RELEASE
if [ $? -ne 0 ]; then
echo "Failed $a:$RELEASE"
fi
done
It seems all container are now correct, at least from my test.
*** Bug 1807612 has been marked as a duplicate of this bug. *** openstack overcloud container image prepare --namespace=registry.access.redhat.com/rhosp13 --push-destination={--PUSH LOCATION--}:8787 --prefix=openstack- --tag-from-label {version}-{release} --output-env-file=/home/stack/templates/overcloud_images-{--DATE--}.yaml --output-images-file /home/stack/local_registry_images-{--DATE--}.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml --set ceph_namespace=registry.access.redhat.com/rhceph --set ceph_image=rhceph-3-rhel7 --tag-from-label {version}-{release} -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovn-dvr-ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/barbican.yaml
i receive the following error message;
Image registry.access.redhat.com/rhosp13/openstack-aodh-evaluator has no tag 13.0-107.
I just tried the same command as the customer and I didn't hit the issue:
~~~
[stack@undercloud-0-rhosp13 ~]$ openstack overcloud container image prepare --namespace=registry.access.redhat.com/rhosp13 --push-destination=192.0.2.1:8787 --prefix=openstack- --tag-from-label {version}-{release} --output-env-file=/home/stack/overcloud_images.yaml --output-images-file /home/stack/local_registry_images-{--DATE--}.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml --set ceph_namespace=registry.access.redhat.com/rhceph --set ceph_image=rhceph-3-rhel7 --tag-from-label {version}-{release} -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovn-dvr-ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/barbican.yaml
container_images:
- imagename: registry.access.redhat.com/rhosp13/openstack-aodh-api:13.0-107
push_destination: 192.0.2.1:8787
- imagename: registry.access.redhat.com/rhosp13/openstack-aodh-evaluator:13.0-107
push_destination: 192.0.2.1:8787
[...]
~~~
My current customer is in EMEA / UK ... Is it possible that their mirrors / CDNs / etc are catching up / broken ?
Customer updated the case with this: ~~~ I have re-run the command this morning and it has worked as expected. When we looked at the container registry via https://catalog.redhat.com/software/containers/explore we noticed that the containers we were having issues with had only been updated a couple of hours prior, so maybe the sync between registry.redhat.com and registry.redhat.io takes a while. ~~~ Possible idea for tripleo upstream to make it more bullet proof working with the assumption that it is a data race with the registry, that the :<tag> fetched form the registry is not on the RepoTags list that skopeo inspect gives so the situation I observed the first time it was reported was that the image available via inspection had a <version>-<release> tag that was fetchable but not in the tags list $ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq ".Labels.release" "100.1582098610" $ curl -L https://registry.access.redhat.com/v2/rhosp13/openstack-aodh-notifier/tags/list {"name": "rhosp13/openstack-aodh-notifier", "tags": ["13.0-85", "13.0-86", "13.0-83", "13.0-100", "13.0", "13.0-56", "13.0-55", "13.0-76", "13.0-72", "13.0-39", "13.0-96", "13.0-100.1580118203", "13.0-62.1543534128", "13.0-68", "13.0-72.1557945115", "13.0-46", "13.0-62", "13.0-60", "13.0-61", "13.0-66", "13.0-64", "13.0-68.1554788878", "latest"]} So the idea would be to implement a fallback instead of failing hard of inspecting the desired image directly so I think it was trying to use RepoTags data from skopeo inspect or something like it to check to see if image was available and failing because it was not in the list. Fallback would be to then attempt to skopeo inspect the explicit image with tag we want to see if it was available and valid, instead of just failing because of inconsistency. skopeo inspect docker://registry.redhat.io/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags [ "13.0-85", "13.0-86", "13.0-83", "13.0-100", "13.0-109", "13.0", "13.0-56", "13.0-55", "13.0-76", "13.0-72", "13.0-39", "13.0-100.1582098610", "13.0-96", "13.0-100.1580118203", "13.0-62.1543534128", "13.0-68", "13.0-72.1557945115", "13.0-46", "13.0-62", "13.0-60", "13.0-61", "13.0-66", "13.0-64", "13.0-68.1554788878", "latest" ] [root@nps ~]# skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags [ "13.0-85", "13.0-86", "13.0-83", "13.0-100", "13.0-109", "13.0", "13.0-56", "13.0-55", "13.0-76", "13.0-72", "13.0-39", "13.0-100.1582098610", "13.0-96", "13.0-100.1580118203", "13.0-62.1543534128", "13.0-68", "13.0-72.1557945115", "13.0-46", "13.0-62", "13.0-60", "13.0-61", "13.0-66", "13.0-64", "13.0-68.1554788878", "latest" ] [root@nps ~]# skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq ".Labels.release" "109" [root@nps ~]# skopeo inspect docker://registry.redhat.io/rhosp13/openstack-aodh-notifier:latest | jq ".Labels.release" "109" source /home/stack/stackrc && openstack overcloud container image prepare --namespace=10.XX.XX.XX:8787/rhosp13 --push-destination="10.XX.XX.XX":8787 --prefix=openstack- --tag-from-label {version}-{release} -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/neutron-sriov.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/neutron-ovs-dpdk.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/cinder-backup.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/collectd.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/fluentd.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sensu-client.yaml -r /home/stack/templates/roles_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml --set ceph_namespace=10.XX.XX.XX:8787/rhceph --set ceph_image=rhceph-3-rhel7 --tag-from-label {version}-{release} --output-env-file=/home/stack/templates/env_files/overcloud_images.yaml --output-images-file /home/stack/local_registry_images.yaml
Error:
Image 10.XX.XX.XX:8787/rhosp13/openstack-aodh-notifier has no tag 13.0-100.1582098610.
issue was identified as possible publishing issue, and has since been resolved $ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq ".Labels.release" "115" $ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags | grep 115 "13.0-115", "13.0-72.1557945115", Additional verification by impacted parties would be good just need to run the above skopeo inspect commands from your system kholtz dhill afariasa Can you verify that this is resolved now? Tested this after yesterday's release of new containers: $ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq ".Labels.release" "120" $ skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest | jq .RepoTags | grep 120 "13.0-120", Everything seems to be in order. Marking VERIFIED. OSP13 support officially ended on 27 June 2023 |
Description of problem: Running: openstack overcloud container image prepare \ --namespace=registry.access.redhat.com/rhosp13 \ --push-destination=192.168.24.1:8787 \ --prefix=openstack- \ --tag-from-label {version}-{release} \ --output-env-file=env.yaml \ --output-images-file=images.yaml Is failing with: Image registry.access.redhat.com/rhosp13/openstack-aodh-notifier has no tag 13.0-100.1582098610. Available tags: 13.0-85, 13.0-86, 13.0-83, 13.0-100, 13.0, 13.0-56, 13.0-55, 13.0-76, 13.0-72, 13.0-39, 13.0-96, 13.0-100.1580118203, 13.0-62.1543534128, 13.0-68, 13.0-72.1557945115, 13.0-46, 13.0-62, 13.0-60, 13.0-61, 13.0-66, 13.0-64, 13.0-68.1554788878, latest And indeed, running: skopeo inspect docker://registry.access.redhat.com/rhosp13/openstack-aodh-notifier:latest Shows that while the "release" label is "100.1582098610", there is no matching tag in RepoTags. There is some indication this may only be affecting *some* of the registry servers; conversations on internal irc suggest that some folks were hitting this and some were not.