Bug 2227676

Is it this: https://bugzilla.redhat.com/show_bug.cgi?id=2142951 ? If so please mark as duplicate.

@Takashi  Sure, assigned it to me.

Test failed.

openstack-tripleo-heat-templates-11.6.1-2.20230808225213.9adcac6.el8ost.noarch
Deploy glance with cache enabled.

1.glance-conf:
------------
[DEFAULT]
image_member_quota=128
show_image_direct_url=True
show_multiple_locations=True
enable_v2_api=True
node_staging_uri=file:///var/lib/glance/staging
enabled_import_methods=[web-download]
bind_host=172.17.1.95
bind_port=9292
workers=4
enabled_backends=default_backend:rbd
image_cache_max_size=10737418240
image_cache_stall_time=86400
image_cache_dir=/var/lib/glance/image-cache
registry_host=0.0.0.0
debug=True
log_file=/var/log/glance/api.log
log_dir=/var/log/glance
transport_url=rabbit://guest:YoHgDTqqXbLC0rsSKBAjo1WkA.redhat.local:5672,guest:YoHgDTqqXbLC0rsSKBAjo1WkA.redhat.local:5672,guest:YoHgDTqqXbLC0rsSKBAjo1WkA.redhat.local:5672/?ssl=0
cache_prefetcher_interval=300
enable_v1_api=False
[cinder]
[cors]
[database]
connection=mysql+pymysql://glance:Jre4yOvrHzWGcO8XRlgqiEkIS.1.84/glance?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo
[file]
[glance.store.http.store]
[glance.store.rbd.store]
[glance.store.sheepdog.store]
[glance.store.swift.store]
[glance.store.vmware_datastore.store]
[glance_store]
default_backend=default_backend
os_region_name=regionOne
[image_format]
[keystone_authtoken]
www_authenticate_uri=http://10.0.0.103:5000
region_name=regionOne
memcached_servers=controller-0.internalapi.redhat.local:11211,controller-1.internalapi.redhat.local:11211,controller-2.internalapi.redhat.local:11211
memcache_use_advanced_pool=True
auth_type=password
auth_url=http://172.17.1.84:5000
username=glance
password=Jre4yOvrHzWGcO8XRlgqiEkIS
user_domain_name=Default
project_name=service
project_domain_name=Default
[oslo_concurrency]
lock_path=/var/lib/glance/tmp



2. Api cron healthcheck
------------------------
[heat-admin@controller-0 ~]$ systemctl status tripleo_glance_api_cron_healthcheck.service                                                                                                                                                    
● tripleo_glance_api_cron_healthcheck.service - glance_api_cron healthcheck
   Loaded: loaded (/etc/systemd/system/tripleo_glance_api_cron_healthcheck.service; disabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2023-10-05 10:13:10 UTC; 55s ago
  Process: 221326 ExecStart=/usr/bin/podman exec --user root glance_api_cron /usr/share/openstack-tripleo-common/healthcheck/cron glance (code=exited, status=0/SUCCESS)                                                                     
 Main PID: 221326 (code=exited, status=0/SUCCESS)


3.
[heat-admin@controller-0 ~]$ sudo podman ps|grep glance                                                                                                                                                                                      
21e1658033b2  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-glance-api:16.2_20230925.1                  kolla_start           About an hour ago  Up About an hour ago          glance_api                                
32e131a6da70  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-glance-api:16.2_20230925.1                  kolla_start           About an hour ago  Up About an hour ago          glance_api_cron  



4.Crontab check
---------------
[heat-admin@controller-0 ~]$ sudo podman exec -it glance_api_cron crontab -l
no crontab for root
WARN[0000] Error resizing exec session c8f76a7a5bd9df1289a85ea7f414baf26ed7a074f2658fc4fe2d8da331722e93: could not open ctl file for terminal resize for container 32e131a6da70622303d6a49c4f1e7f66bf33cbd9548cda36bb91febb5bb57962: open /var/lib/containers/storage/overlay-containers/32e131a6da70622303d6a49c4f1e7f66bf33cbd9548cda36bb91febb5bb57962/userdata/c8f76a7a5bd9df1289a85ea7f414baf26ed7a074f2658fc4fe2d8da331722e93/ctl: no such device or address 
[heat-admin@controller-0 ~]$ sudo podman exec -it glance_api_cron /bin/bash
[root@controller-0 /]# crontab -l
no crontab for root


5.Create image and cache image in controller
----------------------------------------------
bash-4.4$ glance-cache-manage --host=192.168.24.43 queue-image 455f343c-c7db-4484-bdcd-3f386b22ba18
Queue image 455f343c-c7db-4484-bdcd-3f386b22ba18 for caching? [y/N] y
Failed to queue the specified image for caching. Got error:
[Errno 111] Connection refused
bash-4.4$

Max, I'm moving this back to ON_QA because of issues in your verification process.

First, the bug we're fixing is one in which the glance cache is NOT enabled prior to the FFU. In that situation, glance's cron job is not supposed to be enabled after the FFU. The problem (the bug) is that the cron job was created, and was failing because it had nothing to do because the cache isn't enabled. 

For this BZ, I envision the test procedure to do something like this:

- Deploy OSP-13 with glance cache disabled.
- FFU to 16.2 and verify there's no glance_api_cron pod running at all.

Separate from verifying the BZ, I want to note the crontab check you ran in item 4. is not correct. If you look carefully at the tripleo_glance_api_cron_healthcheck.service you will see it runs this command:

==> ExecStart=/usr/bin/podman exec --user root glance_api_cron /usr/share/openstack-tripleo-common/healthcheck/cron glance

Note the last argument is "glance". That gets passed into the /usr/share/openstack-tripleo-common/healthcheck/cron, and it represents the user associated with the cron job. In this case, the cron job runs as the glance user, not root. If you want to see what I mean, try this:

$ sudo podman exec -it glance_api_cron crontab -u glance -l

Just be sure to do that when glance cache is enabled (but verify this BZ with it disabled).

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.2.6 (Train) bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6307