Bug 1949630

Summary: Container images without a subdirectory in the namespace do not get included in _catalog
Product: Red Hat OpenStack Reporter: Direct Docs Feedback <ddf-bot>
Component: openstack-tripleo-commonAssignee: Adriano Petrich <apetrich>
Status: CLOSED ERRATA QA Contact: David Rosenfeld <drosenfe>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: dmacpher, jslagle, mburns, mgeary, slinaber
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: openstack-tripleo-common-11.4.1-1.20211201113403.75bd92a.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-24 10:59:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Direct Docs Feedback 2021-04-14 17:24:46 UTC
when push_destination: true 
and using satellite to configure undercloud, the command sudo openstack tripleo container image list does not show any images. 

Reported by: cbsproject

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/director_installation_and_usage/preparing-for-director-installation#annotations:e033c0ee-046e-4b00-be6e-9b066267e9ac

Comment 1 Dan Macpherson 2021-06-17 15:54:45 UTC
Need to test out the push_destination functionality with satellite to verify this issue.

Comment 2 Dan Macpherson 2021-11-04 04:44:02 UTC
So I've done some investigating and it seems like this is an issue with how the _catalog file is generated on image-serve.

I performed a test where only the keystone image was hosted on Satellite and performed an undercloud installation. The undercloud installed successfully and the keystone container was pulled from the undercloud:

$ sudo podman ps -f name=keystone
CONTAINER ID  IMAGE                                                                              COMMAND      CREATED         STATUS             PORTS  NAMES
cada215a3df6  ccsosp-undercloud.ctlplane.localdomain:8787/acme-osp16_1_containers-keystone:16.1  kolla_start  44 minutes ago  Up 44 minutes ago         keystone

Note that the keystone image is using the Satellite naming convention for images.

So with push_destination: true, the still gets pulled from the Satellite to the undercloud and then pulled from image-serve during installation. However, if you run openstack tripleo container image list, the image doesn't appear. The other images do appear though:

$ sudo openstack tripleo container image list
+-----------------------------------------------------------------------------------------------------------+
| Image Name                                                                                                |
+-----------------------------------------------------------------------------------------------------------+
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-placement-api:16.1             |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-swift-account:16.1             |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-mistral-engine:16.1            |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-qdrouterd:16.1                 |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-nova-api:16.1                  |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-mistral-api:16.1               |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-swift-proxy-server:16.1        |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-rabbitmq:16.1                  |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-heat-engine:16.1               |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-nova-conductor:16.1            |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-heat-api:16.1                  |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-nova-compute-ironic:16.1       |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-ironic-neutron-agent:16.1      |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-server:16.1            |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.1        |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-mariadb:16.1                   |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-rsyslog:16.1                   |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-zaqar-wsgi:16.1                |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-glance-api:16.1                |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-swift-container:16.1           |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-ironic-conductor:16.1          |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-swift-object:16.1              |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.1          |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-cron:16.1                      |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-ironic-inspector:16.1          |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-ironic-api:16.1                |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-haproxy:16.1                   |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-iscsid:16.1                    |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.1 |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-keepalived:16.1                |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-mistral-executor:16.1          |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-nova-scheduler:16.1            |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-memcached:16.1                 |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-ironic-pxe:16.1                |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-tempest:16.1                   |
| docker://ccsosp-undercloud.ctlplane.localdomain:8787/rhosp-rhel8/openstack-mistral-event-engine:16.1      |
+-----------------------------------------------------------------------------------------------------------+

And using curl to view the catalog on image-serve shows now keystone image:

curl http://ccsosp-undercloud.ctlplane.localdomain:8787/v2/_catalog | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1399  100  1399    0     0  1366k      0 --:--:-- --:--:-- --:--:-- 1366k
{
  "repositories": [
    "rhosp-rhel8/openstack-placement-api",
    "rhosp-rhel8/openstack-swift-account",
    "rhosp-rhel8/openstack-mistral-engine",
    "rhosp-rhel8/openstack-qdrouterd",
    "rhosp-rhel8/openstack-nova-api",
    "rhosp-rhel8/openstack-mistral-api",
    "rhosp-rhel8/openstack-swift-proxy-server",
    "rhosp-rhel8/openstack-rabbitmq",
    "rhosp-rhel8/openstack-heat-engine",
    "rhosp-rhel8/openstack-nova-conductor",
    "rhosp-rhel8/openstack-heat-api",
    "rhosp-rhel8/openstack-nova-compute-ironic",
    "rhosp-rhel8/openstack-ironic-neutron-agent",
    "rhosp-rhel8/openstack-neutron-server",
    "rhosp-rhel8/openstack-neutron-dhcp-agent",
    "rhosp-rhel8/openstack-mariadb",
    "rhosp-rhel8/openstack-rsyslog",
    "rhosp-rhel8/openstack-zaqar-wsgi",
    "rhosp-rhel8/openstack-glance-api",
    "rhosp-rhel8/openstack-swift-container",
    "rhosp-rhel8/openstack-ironic-conductor",
    "rhosp-rhel8/openstack-swift-object",
    "rhosp-rhel8/openstack-neutron-l3-agent",
    "rhosp-rhel8/openstack-cron",
    "rhosp-rhel8/openstack-ironic-inspector",
    "rhosp-rhel8/openstack-ironic-api",
    "rhosp-rhel8/openstack-haproxy",
    "rhosp-rhel8/openstack-iscsid",
    "rhosp-rhel8/openstack-neutron-openvswitch-agent",
    "rhosp-rhel8/openstack-keepalived",
    "rhosp-rhel8/openstack-mistral-executor",
    "rhosp-rhel8/openstack-nova-scheduler",
    "rhosp-rhel8/openstack-memcached",
    "rhosp-rhel8/openstack-ironic-pxe",
    "rhosp-rhel8/openstack-tempest",
    "rhosp-rhel8/openstack-mistral-event-engine"
  ]
}

But skopeo does show an image is available:

$ skopeo inspect --tls-verify=false docker://ccsosp-undercloud.ctlplane.localdomain:8787/acme-osp16_1_containers-keystone:16.1
{
    "Name": "ccsosp-undercloud.ctlplane.localdomain:8787/acme-osp16_1_containers-keystone",
    "Digest": "sha256:8e5c6e4865e0ab9d45827ca21ea7a97686669183bfefa95518d7b0f00ea019b7",
    "RepoTags": [
        "16.1"
    ],
    "Created": "2021-10-13T07:48:07.421081Z",
    "DockerVersion": "1.13.1",
    "Labels": {
        "architecture": "x86_64",
        "batch": "16.1_20210616.1",
        "build-date": "2021-10-13T07:47:11.254752",
        "com.redhat.build-host": "cpt-1007.osbs.prod.upshift.rdu2.redhat.com",
...

And the keystone image directory does appear to be in the image-serve directory:
$ sudo ls /var/lib/image-serve/v2/
acme-osp16_1_containers-keystone  _catalog  index.json	rhosp-rhel8

The only difference between the keystone image and the other images is that (due to the Satellite naming convention) keystone isn't using a subdirectory as a namespace while the other images are using rhosp-rhel8.

My guess is there's something in build_catalog() from the tripleo-common [1] that might be causing any images without a subdirectory in the namespace to not get included in the catalog.

Switching this BZ over to tripleo-common.

[1] https://github.com/openstack/tripleo-common/blob/564e46ece866d07c93809ecc9309e26c03475a09/tripleo_common/image/image_export.py#L380

Comment 4 Alex Schultz 2021-11-08 19:27:29 UTC
This should have been fixed via https://review.opendev.org/c/openstack/tripleo-common/+/775107 though it might be in 16.2

Comment 14 errata-xmlrpc 2022-03-24 10:59:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.8 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0986