Bug 1790755

Summary:	[RFE] Support for Glance image pre-cache
Product:	Red Hat OpenStack	Reporter:	Giulio Fidente <gfidente>
Component:	openstack-glance	Assignee:	Cyril Roelandt <cyril>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Mike Abrams <mabrams>
Severity:	medium	Docs Contact:	RHOS Documentation Team <rhos-docs>
Priority:	medium
Version:	16.0 (Train)	CC:	acanan, akekane, athomas, djuran, eglynn, gcharot, lmarsh, mabrams, ndeevy, nlevinki, pgrist, spower, tberry
Target Milestone:	Alpha	Keywords:	FutureFeature, TestOnly, Triaged
Target Release:	16.2 (Train on RHEL 8.4)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-glance-19.0.1-0.20191030153439.6e3ced8.el8ost	Doc Type:	Enhancement
Doc Text:		Story Points:	---
Clone Of:	1706896	Environment:
Last Closed:	2021-09-22 10:07:54 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1706896, 1757310, 1859289
Bug Blocks:	1710557, 1765040

Comment 9 Gregory Charot 2020-03-18 15:22:26 UTC

Moving to 16.1.1, same target as the playbook RFEs.

Comment 13 Lon Hohberger 2020-07-29 10:47:10 UTC

According to our records, this should be resolved by openstack-glance-19.0.3-0.20200421223423.3e16bc7.el8ost.  This build is available now.

Comment 15 Cyril Roelandt 2021-02-15 19:21:48 UTC

Though not strictly required, I merged https://code.engineering.redhat.com/gerrit/#/c/227834/ . It disables all the functional/unit tests related to registry, and will make it easier to merge future backports.

Comment 19 Mike Abrams 2021-05-18 04:19:19 UTC

=== verify fixed-in versions present:

(undercloud) [stack@undercloud-0 ~]$ rhos-release -L
Installed repositories (rhel-8.4):
  16.2
  ceph-4
  ceph-osd-4
  rhel-8.4
(undercloud) [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed 
16.2  -p RHOS-16.2-RHEL-8-20210420.n.0
(undercloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| 44fefa98-6485-42c7-8b34-5c015ef37823 | controller-0 | ACTIVE | ctlplane=192.168.24.7  | overcloud-full | controller |
| 26df0fd8-8875-47bb-9d50-afabb8e6f217 | controller-1 | ACTIVE | ctlplane=192.168.24.6  | overcloud-full | controller |
| 17c4a0ff-f96f-457e-afa3-f7430fcedfd1 | controller-2 | ACTIVE | ctlplane=192.168.24.22 | overcloud-full | controller |
| 52439ac1-e830-4c35-9762-c831e86b3bb2 | compute-0    | ACTIVE | ctlplane=192.168.24.29 | overcloud-full | compute    |
| dd864720-a0ea-4937-8631-9dffffa874cb | compute-1    | ACTIVE | ctlplane=192.168.24.14 | overcloud-full | compute    |
| f06db815-7f88-4fcf-9dfb-1ba43e861d3f | ceph-1       | ACTIVE | ctlplane=192.168.24.37 | overcloud-full | ceph       |
| 745e865b-d8d1-47b5-9a0a-0c1388606518 | ceph-2       | ACTIVE | ctlplane=192.168.24.25 | overcloud-full | ceph       |
| 09bb701e-c44e-452d-a533-1dab7f2d0c30 | ceph-0       | ACTIVE | ctlplane=192.168.24.51 | overcloud-full | ceph       |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
(undercloud) [stack@undercloud-0 ~]$ for i in 7 6 22; do echo "=== $i"; ssh -t heat-admin.24.$i "sudo podman exec -it -u root glance_api sh -c 'rpm -qa openstack-glance'"; done
=== 7
Warning: Permanently added '192.168.24.7' (ECDSA) to the list of known hosts.
openstack-glance-19.0.5-2.20210406124926.b3de9da.el8ost.1.noarch
Connection to 192.168.24.7 closed.
=== 6
Warning: Permanently added '192.168.24.6' (ECDSA) to the list of known hosts.
openstack-glance-19.0.5-2.20210406124926.b3de9da.el8ost.1.noarch
Connection to 192.168.24.6 closed.
=== 22
Warning: Permanently added '192.168.24.22' (ECDSA) to the list of known hosts.
openstack-glance-19.0.5-2.20210406124926.b3de9da.el8ost.1.noarch
Connection to 192.168.24.22 closed.
(undercloud) [stack@undercloud-0 ~]$

Comment 20 Mike Abrams 2021-05-19 08:20:32 UTC

verification:

=== upload image

(overcloud) [stack@undercloud-0 ~]$ . ./overcloudrc 
(overcloud) [stack@undercloud-0 ~]$ glance image-list
+----+------+
| ID | Name |
+----+------+
+----+------+
(overcloud) [stack@undercloud-0 ~]$ glance image-create --name ciros-test-001 --disk-format qcow2 --container-format bare --file ./cirros-0.3.5-i386-disk.img 
+------------------+----------------------------------------------------------------------------------+
| Property         | Value                                                                            |
+------------------+----------------------------------------------------------------------------------+
| checksum         | 7316af7358dd32ca1956d72ac2c9e147                                                 |
| container_format | bare                                                                             |
| created_at       | 2021-05-19T07:53:57Z                                                             |
| direct_url       | swift+config://ref1/glance/fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce                  |
| disk_format      | qcow2                                                                            |
| id               | fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce                                             |
| min_disk         | 0                                                                                |
| min_ram          | 0                                                                                |
| name             | ciros-test-001                                                                   |
| os_hash_algo     | sha512                                                                           |
| os_hash_value    | 734c9281adf72b9947eb9ab85f5e9db0fe388b742ebb68469d4e87d17065e7a39b501c184e60913c |
|                  | 79ec9e79042992024a2063b906c66a5c37735626f8f14bae                                 |
| os_hidden        | False                                                                            |
| owner            | 1ff28d5263bc472ab17aab6fffcbab96                                                 |
| protected        | False                                                                            |
| size             | 12528640                                                                         |
| status           | active                                                                           |
| stores           | default_backend                                                                  |
| tags             | []                                                                               |
| updated_at       | 2021-05-19T07:53:59Z                                                             |
| virtual_size     | Not available                                                                    |
| visibility       | shared                                                                           |
+------------------+----------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$

=== verify prefetcher interval is set in glance-api.conf on the containers to 300 seconds:

(overcloud) [stack@undercloud-0 ~]$ for i in 41 49 53; do echo "=== $i"; ssh -t heat-admin.24.$i "sudo podman exec -it -u root glance_api sh -c 'grep ^cache_prefetcher_interval /etc/glance/glance-api.conf| grep -v ^# ' 2> /dev/null"; done 
=== 41
Warning: Permanently added '192.168.24.41' (ECDSA) to the list of known hosts.
cache_prefetcher_interval=300
Connection to 192.168.24.41 closed.
=== 49
Warning: Permanently added '192.168.24.49' (ECDSA) to the list of known hosts.
cache_prefetcher_interval=300
Connection to 192.168.24.49 closed.
=== 53
Warning: Permanently added '192.168.24.53' (ECDSA) to the list of known hosts.
cache_prefetcher_interval=300
Connection to 192.168.24.53 closed.
(overcloud) [stack@undercloud-0 ~]$

=== gather glance ip's:
[heat-admin@controller-0 ~]$ sudo grep -A 10 '^listen glance_api' /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg
listen glance_api
  bind 10.0.0.126:9292 transparent
  bind 172.17.1.36:9292 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  option httpchk GET /healthcheck
  server controller-0.internalapi.redhat.local 172.17.1.24:9292 check fall 5 inter 2000 rise 2
  server controller-1.internalapi.redhat.local 172.17.1.92:9292 check fall 5 inter 2000 rise 2
  server controller-2.internalapi.redhat.local 172.17.1.71:9292 check fall 5 inter 2000 rise 2
[heat-admin@controller-0 ~]$

=== confirm no queued images yet:

[heat-admin@controller-0 ~]$ . ./overcloudrc 
(overcloud) [heat-admin@controller-0 ~]$ for i in 24 92 71; do echo "=== 172.17.1.$i"; glance-cache-manage --host 172.17.1.$i list-queued; done
=== 172.17.1.24
No queued images.
=== 172.17.1.92
No queued images.
=== 172.17.1.71
No queued images.
(overcloud) [heat-admin@controller-0 ~]$

=== queue image and allow time to prefetch:

(undercloud) [stack@undercloud-0 ~]$ . ./overcloudrc 
(overcloud) [stack@undercloud-0 ~]$ glance image-list
+--------------------------------------+----------------+
| ID                                   | Name           |
+--------------------------------------+----------------+
| fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce | ciros-test-001 |
+--------------------------------------+----------------+
(overcloud) [stack@undercloud-0 ~]$ ssh -t heat-admin.24.53
Warning: Permanently added '192.168.24.53' (ECDSA) to the list of known hosts.
Last login: Wed May 19 08:09:28 2021 from 192.168.24.1
[heat-admin@controller-0 ~]$ . ./overcloudrc 
(overcloud) [heat-admin@controller-0 ~]$ for i in 24 92 71; do echo "=== 172.17.1.$i"; glance-cache-manage --host 172.17.1.$i queue-image fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce; done
=== 172.17.1.24
Queue image fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce for caching? [y/N] y
=== 172.17.1.92
Queue image fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce for caching? [y/N] y
=== 172.17.1.71
Queue image fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce for caching? [y/N] y
(overcloud) [heat-admin@controller-0 ~]$ 
(overcloud) [heat-admin@controller-0 ~]$ sleep 360
(overcloud) [heat-admin@controller-0 ~]$

=== list cached image:

(overcloud) [heat-admin@controller-0 ~]$ for i in 24 92 71; do echo "=== 172.17.1.$i"; glance-cache-manage --host 172.17.1.$i list-cached; done
=== 172.17.1.24
Found 1 cached images...
+--------------------------------------+----------------------------+----------------------------+----------+------+
|                  ID                  |    Last Accessed (UTC)     |    Last Modified (UTC)     |     Size | Hits |
+--------------------------------------+----------------------------+----------------------------+----------+------+
| fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce | 2021-05-19T08:16:12.477724 | 2021-05-19T08:16:12.477724 | 12528640 |    0 |
+--------------------------------------+----------------------------+----------------------------+----------+------+
=== 172.17.1.92
Found 1 cached images...
+--------------------------------------+----------------------------+----------------------------+----------+------+
|                  ID                  |    Last Accessed (UTC)     |    Last Modified (UTC)     |     Size | Hits |
+--------------------------------------+----------------------------+----------------------------+----------+------+
| fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce | 2021-05-19T08:16:00.191832 | 2021-05-19T08:16:00.191832 | 12528640 |    0 |
+--------------------------------------+----------------------------+----------------------------+----------+------+
=== 172.17.1.71
Found 1 cached images...
+--------------------------------------+----------------------------+----------------------------+----------+------+
|                  ID                  |    Last Accessed (UTC)     |    Last Modified (UTC)     |     Size | Hits |
+--------------------------------------+----------------------------+----------------------------+----------+------+
| fdd9c31b-6d0d-4971-8f8f-cbf7924f14ce | 2021-05-19T08:16:02.183637 | 2021-05-19T08:16:02.183637 | 12528640 |    0 |
+--------------------------------------+----------------------------+----------------------------+----------+------+
(overcloud) [heat-admin@controller-0 ~]$

VERIFIED