Bug 2309724 - race in tripleo_container_manage
Summary: race in tripleo_container_manage
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: tripleo-ansible
Version: 17.1 (Wallaby)
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: z4
: 17.1
Assignee: Lukas Bezdicka
QA Contact: Archana Singh
URL:
Whiteboard:
: 2316429 (view as bug list)
Depends On:
Blocks: 2316429
TreeView+ depends on / blocked
 
Reported: 2024-09-04 14:45 UTC by Lukas Bezdicka
Modified: 2024-11-25 20:59 UTC (History)
8 users (show)

Fixed In Version: tripleo-ansible-3.3.1-17.1.20241015110802.8debef3.el9osttrunk tripleo-ansible-3.3.1-17.1.20241015113746.8debef3.el8osttrunk
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2316429 (view as bug list)
Environment:
Last Closed: 2024-11-21 09:42:32 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-32754 0 None None None 2024-09-04 14:46:22 UTC
Red Hat Product Errata RHBA-2024:9974 0 None None None 2024-11-21 09:42:36 UTC

Description Lukas Bezdicka 2024-09-04 14:45:44 UTC 
When tripleo_container_manage starts containers it uses PodmanManage. PodmanManage checks if container is running and stops it before running container start. This would work fine on clean deploy but tripleo uses systemd services for containers. Sometimes systemd reacts facter than PodmanManage and during update or upgrade starts back container before the new one is spawned. This results in ansible failure. There is higher chance of this happening on bigger environments and those are also costly to rerun.
Comment 11 Lukas Bezdicka 2024-10-11 12:08:31 UTC 
Failing QA 

scale up or scale down operations would end with outage on control plane and might even affect dataplane:

[stack@undercloud-0 ~]$ grep "WARNING | Stopped" overcloud_scale_up.log
2024-10-11 11:38:03.403122 |                                      |    WARNING | Stopped service tripleo_memcached
2024-10-11 11:38:03.480668 |                                      |    WARNING | Stopped service tripleo_memcached
2024-10-11 11:42:13.481965 |                                      |    WARNING | Stopped service tripleo_clustercheck
2024-10-11 11:42:13.804133 |                                      |    WARNING | Stopped service tripleo_clustercheck
2024-10-11 11:43:08.230257 |                                      |    WARNING | Stopped service tripleo_nova_virtlogd_wrapper
2024-10-11 11:43:08.231361 |                                      |    WARNING | Stopped service tripleo_ovn_controller
2024-10-11 11:43:08.231820 |                                      |    WARNING | Stopped service tripleo_nova_libvirt
2024-10-11 11:43:08.232339 |                                      |    WARNING | Stopped service tripleo_iscsid
2024-10-11 11:43:28.320755 |                                      |    WARNING | Stopped service tripleo_horizon
2024-10-11 11:43:28.321806 |                                      |    WARNING | Stopped service tripleo_ovn_controller
2024-10-11 11:43:28.322489 |                                      |    WARNING | Stopped service tripleo_iscsid
2024-10-11 11:43:28.323190 |                                      |    WARNING | Stopped service tripleo_keystone
2024-10-11 11:43:28.323881 |                                      |    WARNING | Stopped service tripleo_keystone_cron
2024-10-11 11:43:29.400431 |                                      |    WARNING | Stopped service tripleo_horizon
2024-10-11 11:43:29.401397 |                                      |    WARNING | Stopped service tripleo_ovn_controller
2024-10-11 11:43:29.401897 |                                      |    WARNING | Stopped service tripleo_iscsid
2024-10-11 11:43:29.402458 |                                      |    WARNING | Stopped service tripleo_keystone
2024-10-11 11:43:29.402859 |                                      |    WARNING | Stopped service tripleo_keystone_cron
2024-10-11 11:44:31.207074 |                                      |    WARNING | Stopped service tripleo_nova_virtlogd_wrapper
2024-10-11 11:44:31.208801 |                                      |    WARNING | Stopped service tripleo_ovn_controller
2024-10-11 11:44:31.210098 |                                      |    WARNING | Stopped service tripleo_nova_virtsecretd
2024-10-11 11:44:31.211791 |                                      |    WARNING | Stopped service tripleo_iscsid
2024-10-11 11:44:31.212445 |                                      |    WARNING | Stopped service tripleo_nova_virtnodedevd
2024-10-11 11:44:31.214456 |                                      |    WARNING | Stopped service tripleo_nova_virtstoraged
2024-10-11 11:44:31.215044 |                                      |    WARNING | Stopped service tripleo_nova_virtqemud
2024-10-11 11:44:31.216242 |                                      |    WARNING | Stopped service tripleo_nova_virtproxyd
2024-10-11 11:47:32.366667 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_north_db_server
2024-10-11 11:47:32.502762 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_north_db_server
2024-10-11 11:47:42.792046 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_south_db_server
2024-10-11 11:47:43.048683 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_south_db_server
2024-10-11 11:47:53.803595 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_northd
2024-10-11 11:47:54.293897 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_northd
2024-10-11 11:48:01.040551 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:48:02.578835 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:48:03.119513 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:48:10.502090 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:48:10.509531 |                                      |    WARNING | Stopped service tripleo_nova_migration_target
2024-10-11 11:48:10.510536 |                                      |    WARNING | Stopped service tripleo_ovn_metadata_agent
2024-10-11 11:48:10.513469 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:48:10.514130 |                                      |    WARNING | Stopped service tripleo_nova_migration_target
2024-10-11 11:48:10.514617 |                                      |    WARNING | Stopped service tripleo_ovn_metadata_agent
2024-10-11 11:49:57.094538 |                                      |    WARNING | Stopped service tripleo_cinder_api_cron
2024-10-11 11:49:57.096113 |                                      |    WARNING | Stopped service tripleo_heat_api_cron
2024-10-11 11:49:57.096584 |                                      |    WARNING | Stopped service tripleo_heat_api_cfn
2024-10-11 11:49:57.097023 |                                      |    WARNING | Stopped service tripleo_cinder_api
2024-10-11 11:49:57.097491 |                                      |    WARNING | Stopped service tripleo_heat_api
2024-10-11 11:49:57.097948 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:49:57.098440 |                                      |    WARNING | Stopped service tripleo_nova_conductor
2024-10-11 11:49:57.098883 |                                      |    WARNING | Stopped service tripleo_nova_vnc_proxy
2024-10-11 11:49:57.099343 |                                      |    WARNING | Stopped service tripleo_neutron_api
2024-10-11 11:49:57.099759 |                                      |    WARNING | Stopped service tripleo_cinder_scheduler
2024-10-11 11:49:57.100218 |                                      |    WARNING | Stopped service tripleo_nova_scheduler
2024-10-11 11:49:57.100696 |                                      |    WARNING | Stopped service tripleo_heat_engine
2024-10-11 11:49:57.101146 |                                      |    WARNING | Stopped service tripleo_placement_api
2024-10-11 11:49:57.101579 |                                      |    WARNING | Stopped service tripleo_glance_api_cron
2024-10-11 11:49:57.102029 |                                      |    WARNING | Stopped service tripleo_glance_api
2024-10-11 11:49:57.102562 |                                      |    WARNING | Stopped service tripleo_glance_api_internal
2024-10-11 11:49:57.102995 |                                      |    WARNING | Stopped service tripleo_nova_metadata
2024-10-11 11:49:57.103482 |                                      |    WARNING | Stopped service tripleo_nova_api
2024-10-11 11:49:57.103901 |                                      |    WARNING | Stopped service tripleo_nova_api_cron
2024-10-11 11:49:57.207366 |                                      |    WARNING | Stopped service tripleo_cinder_api_cron
2024-10-11 11:49:57.209149 |                                      |    WARNING | Stopped service tripleo_heat_api_cron
2024-10-11 11:49:57.209671 |                                      |    WARNING | Stopped service tripleo_heat_api
2024-10-11 11:49:57.210124 |                                      |    WARNING | Stopped service tripleo_cinder_api
2024-10-11 11:49:57.210565 |                                      |    WARNING | Stopped service tripleo_heat_api_cfn
2024-10-11 11:49:57.210982 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:49:57.211432 |                                      |    WARNING | Stopped service tripleo_nova_conductor
2024-10-11 11:49:57.211844 |                                      |    WARNING | Stopped service tripleo_neutron_api
2024-10-11 11:49:57.212284 |                                      |    WARNING | Stopped service tripleo_cinder_scheduler
2024-10-11 11:49:57.212698 |                                      |    WARNING | Stopped service tripleo_nova_vnc_proxy
2024-10-11 11:49:57.213136 |                                      |    WARNING | Stopped service tripleo_nova_scheduler
2024-10-11 11:49:57.213564 |                                      |    WARNING | Stopped service tripleo_heat_engine
2024-10-11 11:49:57.213980 |                                      |    WARNING | Stopped service tripleo_placement_api
2024-10-11 11:49:57.214465 |                                      |    WARNING | Stopped service tripleo_glance_api_cron
2024-10-11 11:49:57.214879 |                                      |    WARNING | Stopped service tripleo_glance_api
2024-10-11 11:49:57.215351 |                                      |    WARNING | Stopped service tripleo_glance_api_internal
2024-10-11 11:49:57.215787 |                                      |    WARNING | Stopped service tripleo_nova_api
2024-10-11 11:49:57.216244 |                                      |    WARNING | Stopped service tripleo_nova_metadata
2024-10-11 11:49:57.216728 |                                      |    WARNING | Stopped service tripleo_nova_api_cron
2024-10-11 11:52:16.252390 |                                      |    WARNING | Stopped service tripleo_nova_compute
2024-10-11 11:52:16.871009 |                                      |    WARNING | Stopped service tripleo_nova_compute
Comment 21 errata-xmlrpc 2024-11-21 09:42:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHOSP 17.1.4 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:9974
Comment 22 pweeks 2024-11-25 20:59:12 UTC 
*** Bug 2316429 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.