Moving to 16.1.1, same target as the playbook RFEs.
According to our records, this should be resolved by openstack-glance-19.0.3-0.20200421223423.3e16bc7.el8ost. This build is available now.
Though not strictly required, I merged https://code.engineering.redhat.com/gerrit/#/c/227834/ . It disables all the functional/unit tests related to registry, and will make it easier to merge future backports.
=== verify fixed-in versions present: (undercloud) [stack@undercloud-0 ~]$ rhos-release -L Installed repositories (rhel-8.4): 16.2 ceph-4 ceph-osd-4 rhel-8.4 (undercloud) [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed 16.2 -p RHOS-16.2-RHEL-8-20210420.n.0 (undercloud) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | 44fefa98-6485-42c7-8b34-5c015ef37823 | controller-0 | ACTIVE | ctlplane=192.168.24.7 | overcloud-full | controller | | 26df0fd8-8875-47bb-9d50-afabb8e6f217 | controller-1 | ACTIVE | ctlplane=192.168.24.6 | overcloud-full | controller | | 17c4a0ff-f96f-457e-afa3-f7430fcedfd1 | controller-2 | ACTIVE | ctlplane=192.168.24.22 | overcloud-full | controller | | 52439ac1-e830-4c35-9762-c831e86b3bb2 | compute-0 | ACTIVE | ctlplane=192.168.24.29 | overcloud-full | compute | | dd864720-a0ea-4937-8631-9dffffa874cb | compute-1 | ACTIVE | ctlplane=192.168.24.14 | overcloud-full | compute | | f06db815-7f88-4fcf-9dfb-1ba43e861d3f | ceph-1 | ACTIVE | ctlplane=192.168.24.37 | overcloud-full | ceph | | 745e865b-d8d1-47b5-9a0a-0c1388606518 | ceph-2 | ACTIVE | ctlplane=192.168.24.25 | overcloud-full | ceph | | 09bb701e-c44e-452d-a533-1dab7f2d0c30 | ceph-0 | ACTIVE | ctlplane=192.168.24.51 | overcloud-full | ceph | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ (undercloud) [stack@undercloud-0 ~]$ for i in 7 6 22; do echo "=== $i"; ssh -t heat-admin.24.$i "sudo podman exec -it -u root glance_api sh -c 'rpm -qa openstack-glance'"; done === 7 Warning: Permanently added '192.168.24.7' (ECDSA) to the list of known hosts. openstack-glance-19.0.5-2.20210406124926.b3de9da.el8ost.1.noarch Connection to 192.168.24.7 closed. === 6 Warning: Permanently added '192.168.24.6' (ECDSA) to the list of known hosts. openstack-glance-19.0.5-2.20210406124926.b3de9da.el8ost.1.noarch Connection to 192.168.24.6 closed. === 22 Warning: Permanently added '192.168.24.22' (ECDSA) to the list of known hosts. openstack-glance-19.0.5-2.20210406124926.b3de9da.el8ost.1.noarch Connection to 192.168.24.22 closed. (undercloud) [stack@undercloud-0 ~]$
verification: === upload image (overcloud) [stack@undercloud-0 ~]$ . ./overcloudrc (overcloud) [stack@undercloud-0 ~]$ glance image-list +----+------+ | ID | Name | +----+------+ +----+------+ (overcloud) [stack@undercloud-0 ~]$ glance image-create --name ciros-test-001 --disk-format qcow2 --container-format bare --file ./cirros-0.3.5-i386-disk.img +------------------+----------------------------------------------------------------------------------+ | Property | Value | +------------------+----------------------------------------------------------------------------------+ | checksum | 7316af7358dd32ca1956d72ac2c9e147 | | container_format | bare | | created_at | 2021-05-19T07:53:57Z | | direct_url | swift+config://ref1/glance/fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce | | disk_format | qcow2 | | id | fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce | | min_disk | 0 | | min_ram | 0 | | name | ciros-test-001 | | os_hash_algo | sha512 | | os_hash_value | 734c9281adf72b9947eb9ab85f5e9db0fe388b742ebb68469d4e87d17065e7a39b501c184e60913c | | | 79ec9e79042992024a2063b906c66a5c37735626f8f14bae | | os_hidden | False | | owner | 1ff28d5263bc472ab17aab6fffcbab96 | | protected | False | | size | 12528640 | | status | active | | stores | default_backend | | tags | [] | | updated_at | 2021-05-19T07:53:59Z | | virtual_size | Not available | | visibility | shared | +------------------+----------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ === verify prefetcher interval is set in glance-api.conf on the containers to 300 seconds: (overcloud) [stack@undercloud-0 ~]$ for i in 41 49 53; do echo "=== $i"; ssh -t heat-admin.24.$i "sudo podman exec -it -u root glance_api sh -c 'grep ^cache_prefetcher_interval /etc/glance/glance-api.conf| grep -v ^# ' 2> /dev/null"; done === 41 Warning: Permanently added '192.168.24.41' (ECDSA) to the list of known hosts. cache_prefetcher_interval=300 Connection to 192.168.24.41 closed. === 49 Warning: Permanently added '192.168.24.49' (ECDSA) to the list of known hosts. cache_prefetcher_interval=300 Connection to 192.168.24.49 closed. === 53 Warning: Permanently added '192.168.24.53' (ECDSA) to the list of known hosts. cache_prefetcher_interval=300 Connection to 192.168.24.53 closed. (overcloud) [stack@undercloud-0 ~]$ === gather glance ip's: [heat-admin@controller-0 ~]$ sudo grep -A 10 '^listen glance_api' /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg listen glance_api bind 10.0.0.126:9292 transparent bind 172.17.1.36:9292 transparent mode http http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto http if !{ ssl_fc } http-request set-header X-Forwarded-Port %[dst_port] option httpchk GET /healthcheck server controller-0.internalapi.redhat.local 172.17.1.24:9292 check fall 5 inter 2000 rise 2 server controller-1.internalapi.redhat.local 172.17.1.92:9292 check fall 5 inter 2000 rise 2 server controller-2.internalapi.redhat.local 172.17.1.71:9292 check fall 5 inter 2000 rise 2 [heat-admin@controller-0 ~]$ === confirm no queued images yet: [heat-admin@controller-0 ~]$ . ./overcloudrc (overcloud) [heat-admin@controller-0 ~]$ for i in 24 92 71; do echo "=== 172.17.1.$i"; glance-cache-manage --host 172.17.1.$i list-queued; done === 172.17.1.24 No queued images. === 172.17.1.92 No queued images. === 172.17.1.71 No queued images. (overcloud) [heat-admin@controller-0 ~]$ === queue image and allow time to prefetch: (undercloud) [stack@undercloud-0 ~]$ . ./overcloudrc (overcloud) [stack@undercloud-0 ~]$ glance image-list +--------------------------------------+----------------+ | ID | Name | +--------------------------------------+----------------+ | fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce | ciros-test-001 | +--------------------------------------+----------------+ (overcloud) [stack@undercloud-0 ~]$ ssh -t heat-admin.24.53 Warning: Permanently added '192.168.24.53' (ECDSA) to the list of known hosts. Last login: Wed May 19 08:09:28 2021 from 192.168.24.1 [heat-admin@controller-0 ~]$ . ./overcloudrc (overcloud) [heat-admin@controller-0 ~]$ for i in 24 92 71; do echo "=== 172.17.1.$i"; glance-cache-manage --host 172.17.1.$i queue-image fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce; done === 172.17.1.24 Queue image fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce for caching? [y/N] y === 172.17.1.92 Queue image fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce for caching? [y/N] y === 172.17.1.71 Queue image fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce for caching? [y/N] y (overcloud) [heat-admin@controller-0 ~]$ (overcloud) [heat-admin@controller-0 ~]$ sleep 360 (overcloud) [heat-admin@controller-0 ~]$ === list cached image: (overcloud) [heat-admin@controller-0 ~]$ for i in 24 92 71; do echo "=== 172.17.1.$i"; glance-cache-manage --host 172.17.1.$i list-cached; done === 172.17.1.24 Found 1 cached images... +--------------------------------------+----------------------------+----------------------------+----------+------+ | ID | Last Accessed (UTC) | Last Modified (UTC) | Size | Hits | +--------------------------------------+----------------------------+----------------------------+----------+------+ | fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce | 2021-05-19T08:16:12.477724 | 2021-05-19T08:16:12.477724 | 12528640 | 0 | +--------------------------------------+----------------------------+----------------------------+----------+------+ === 172.17.1.92 Found 1 cached images... +--------------------------------------+----------------------------+----------------------------+----------+------+ | ID | Last Accessed (UTC) | Last Modified (UTC) | Size | Hits | +--------------------------------------+----------------------------+----------------------------+----------+------+ | fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce | 2021-05-19T08:16:00.191832 | 2021-05-19T08:16:00.191832 | 12528640 | 0 | +--------------------------------------+----------------------------+----------------------------+----------+------+ === 172.17.1.71 Found 1 cached images... +--------------------------------------+----------------------------+----------------------------+----------+------+ | ID | Last Accessed (UTC) | Last Modified (UTC) | Size | Hits | +--------------------------------------+----------------------------+----------------------------+----------+------+ | fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce | 2021-05-19T08:16:02.183637 | 2021-05-19T08:16:02.183637 | 12528640 | 0 | +--------------------------------------+----------------------------+----------------------------+----------+------+ (overcloud) [heat-admin@controller-0 ~]$ VERIFIED