Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1919445

Summary: "openstack tripleo container image list" adds extra "tags" to path causing 404 errors
Product: Red Hat OpenStack Reporter: Cody Swanson <cswanson>
Component: openstack-tripleo-commonAssignee: Alex Schultz <aschultz>
Status: CLOSED ERRATA QA Contact: David Rosenfeld <drosenfe>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: aschultz, hbrock, jamsmith, jslagle, mburns, mvalsecc, slinaber
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tripleo-ansible-0.5.1-1.20210319143514.902c3c8.el8ost openstack-tripleo-common-11.4.1-1.20210319144626.75bd92a.el8ost Doc Type: Bug Fix
Doc Text:
This update modifies the registry metadata creator to handle containers with and without namespaces in their URI. On the undercloud you can now manage containers that comply with the following formats: + undercloud_host:port/namespace/container:tag undercloud_host:port/container:tag + Red Hat does not support more complex namespaces, such as undercloud_host:port/name/space/container:tag, when pushing to the undercloud.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-26 13:50:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
file list of the /var/lib/image-serve directory on the undercloud.
none
"openstack --debug tripleo container image list" command output none

Description Cody Swanson 2021-01-22 20:47:58 UTC
Description of problem:

Running "openstack tripleo container image list" on the undercloud host results in 404 errors. The command seems to successfully pull the _catalog from the registry:

http://undercloud.ctlplane.localdomain:8787 "GET /v2/ HTTP/1.1" 200 2
http://undercloud.ctlplane.localdomain:8787/v2/ status code 200
http://undercloud.ctlplane.localdomain:8787 "GET /v2/_catalog HTTP/1.1" 200 5100

But then seems to run into problems when trying to pull the container tags (swift-proxy for example):

http://undercloud.ctlplane.localdomain:8787 "GET /v2/hpe-osp16_1_containers-swift-proxy-server/blobs/tags/list HTTP/1.1" 404 258
http://undercloud.ctlplane.localdomain:8787 "GET /v2/hpe-osp16_1_containers-swift-proxy-server/manifests/tags/list HTTP/1.1" 404 262
http://undercloud.ctlplane.localdomain:8787 "GET /v2/hpe-osp16_1_containers-swift-proxy-server/tags/tags/list HTTP/1.1" 404 257

There seems to be an extra "/tags" added to the URI of each container listed in the _catalog. These extraneous "/tags" are no present when the _catalog is directly queried using curl. 

(undercloud) [stack@undercloud ~]$ curl http://undercloud.ctlplane.localdomain:8787/v2/_catalog | jq .repositories |grep swift-proxy

 "hpe-osp16_1_containers-swift-proxy-server/blobs",
 "hpe-osp16_1_containers-swift-proxy-server/manifests",
 "hpe-osp16_1_containers-swift-proxy-server/tags",

This does not happen on another identical environment where the containers are not prefixed with "hpe-osp16_1_containers". When users follow our director installation and usage documentation for setting up satellite as a container host, creating a content view in satellite adds this container name prefix.

Version-Release number of selected component (if applicable):

openstack-tripleo-common-11.4.1-1.20200914165651.el8ost.noarch Fri Dec  4 14:19:09 2020
openstack-tripleo-common-containers-11.4.1-1.20200914165651.el8ost.noarch Fri Dec  4 14:17:53 2020
openstack-tripleo-heat-templates-11.3.2-1.20200914170156.el8ost.noarch Fri Dec  4 14:19:11 2020
openstack-tripleo-image-elements-10.6.2-0.20200528043425.7dc0fa1.el8ost.noarch Fri Dec  4 14:19:14 2020
openstack-tripleo-puppet-elements-11.2.2-0.20200701163410.432518a.el8ost.noarch Fri Dec  4 14:18:50 2020
openstack-tripleo-validations-11.3.2-1.20200914170825.el8ost.noarch Fri Dec  4 14:18:15 2020
puppet-openstacklib-15.4.1-0.20200403203429.5fdf43c.el8ost.noarch Fri Dec  4 14:18:35 2020
puppet-openstack_extras-15.4.1-0.20200528113453.371931c.el8ost.noarch Fri Dec  4 14:18:27 2020
python3-openstackclient-4.0.1-1.20200817092223.bff556c.el8ost.noarch Fri Dec  4 14:18:57 2020
python3-openstacksdk-0.36.4-0.20200715054250.76d3b29.el8ost.noarch Fri Dec  4 14:17:33 2020
python-openstackclient-lang-4.0.1-1.20200817092223.bff556c.el8ost.noarch Fri Dec  4 14:18:55 2020

How reproducible:

Every time.

Steps to Reproduce:
1. Follow 16.1 director documentation for using satellite as a container mirror. 
2. Install undercloud following 16.1 documentation
3. Run openstack tripleo container image list

Actual results:

Command returns 404's for every container on the undercloud image-serve host. 

Expected results:

Openstack tripleo container image list returns list of images hosted on undercloud.

Additional info:

Full command output and some additional system details are attached to this bug.

Comment 1 Cody Swanson 2021-01-22 20:50:09 UTC
Created attachment 1749877 [details]
file list of the /var/lib/image-serve directory on the undercloud.

Comment 2 Cody Swanson 2021-01-22 20:52:38 UTC
Created attachment 1749879 [details]
"openstack --debug tripleo container image list" command output

Comment 3 Alex Schultz 2021-01-22 22:43:27 UTC
Please provide the container-image-prepare.yaml used during the installation.

Comment 5 Alex Schultz 2021-01-27 15:59:50 UTC
The namespace configuration is bad.

   namespace: rhss.dev.com:5000  


It usually includes something like rhosp-rhel8.

   namespace: rhss.dev.com:5000/rhosp-rhel8


We use the / split names since the namespace is missing the namespace and only provides the host, the container urls end up incorrect.  Given this is a satellite server, both namespace and ceph namespace(s) should likely have a namespace included with the host.

Comment 10 Alex Schultz 2021-01-29 16:00:48 UTC
So I believe this is an issue with assumptions we've made around containers always having a / separating namespace and container name. Under satellite server we do not have an normal namespace as it's just host/containername:tag rather than host/namespace/containername:tag.  The image list is empty when you query it having used push_destination: true and installing the undercloud, while querying the catalog returns the extra directories. At this time, I don't think the `openstack tripleo container image` CRUD commands will work with satellite sourced images.  

Is this preventing a deployment?  If so I would recommend dropping push_destination: true and just using the satellite as the source the deployment across the board.  We'd have to figure out how to fix the image-serve to understand containers that don't have an actual namespace. If it's not breaking deployment, I think we just need to document that you can't use `openstack tripleo container image list` to see the undercloud container registry's contents of satellite sourced containers.

Comment 11 Alex Schultz 2021-01-29 18:14:28 UTC
I think I have a way to fix the catalog. It'll be a bit before it can make it back to 16.1. Please let me know if this is actually causing failures or if it's just related to the cli commands.

Comment 19 Takashi Kajinami 2021-04-23 02:49:05 UTC
*** Bug 1952738 has been marked as a duplicate of this bug. ***

Comment 25 errata-xmlrpc 2021-05-26 13:50:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.6 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2097