Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2309724

Summary:	race in tripleo_container_manage
Product:	Red Hat OpenStack	Reporter:	Lukas Bezdicka <lbezdick>
Component:	tripleo-ansible	Assignee:	Lukas Bezdicka <lbezdick>
Status:	CLOSED ERRATA	QA Contact:	Archana Singh <arcsingh>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	17.1 (Wallaby)	CC:	arcsingh, dhughes, jbadiapa, jpretori, jstransk, mariel, pweeks, tvignaud
Target Milestone:	z4	Keywords:	Triaged
Target Release:	17.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	tripleo-ansible-3.3.1-17.1.20241015110802.8debef3.el9osttrunk tripleo-ansible-3.3.1-17.1.20241015113746.8debef3.el8osttrunk	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2316429 (view as bug list)		Environment:
Last Closed:	2024-11-21 09:42:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2316429

Description Lukas Bezdicka 2024-09-04 14:45:44 UTC

When tripleo_container_manage starts containers it uses PodmanManage. PodmanManage checks if container is running and stops it before running container start. This would work fine on clean deploy but tripleo uses systemd services for containers. Sometimes systemd reacts facter than PodmanManage and during update or upgrade starts back container before the new one is spawned. This results in ansible failure. There is higher chance of this happening on bigger environments and those are also costly to rerun.

Comment 11 Lukas Bezdicka 2024-10-11 12:08:31 UTC

Failing QA 

scale up or scale down operations would end with outage on control plane and might even affect dataplane:

[stack@undercloud-0 ~]$ grep "WARNING | Stopped" overcloud_scale_up.log
2024-10-11 11:38:03.403122 |                                      |    WARNING | Stopped service tripleo_memcached
2024-10-11 11:38:03.480668 |                                      |    WARNING | Stopped service tripleo_memcached
2024-10-11 11:42:13.481965 |                                      |    WARNING | Stopped service tripleo_clustercheck
2024-10-11 11:42:13.804133 |                                      |    WARNING | Stopped service tripleo_clustercheck
2024-10-11 11:43:08.230257 |                                      |    WARNING | Stopped service tripleo_nova_virtlogd_wrapper
2024-10-11 11:43:08.231361 |                                      |    WARNING | Stopped service tripleo_ovn_controller
2024-10-11 11:43:08.231820 |                                      |    WARNING | Stopped service tripleo_nova_libvirt
2024-10-11 11:43:08.232339 |                                      |    WARNING | Stopped service tripleo_iscsid
2024-10-11 11:43:28.320755 |                                      |    WARNING | Stopped service tripleo_horizon
2024-10-11 11:43:28.321806 |                                      |    WARNING | Stopped service tripleo_ovn_controller
2024-10-11 11:43:28.322489 |                                      |    WARNING | Stopped service tripleo_iscsid
2024-10-11 11:43:28.323190 |                                      |    WARNING | Stopped service tripleo_keystone
2024-10-11 11:43:28.323881 |                                      |    WARNING | Stopped service tripleo_keystone_cron
2024-10-11 11:43:29.400431 |                                      |    WARNING | Stopped service tripleo_horizon
2024-10-11 11:43:29.401397 |                                      |    WARNING | Stopped service tripleo_ovn_controller
2024-10-11 11:43:29.401897 |                                      |    WARNING | Stopped service tripleo_iscsid
2024-10-11 11:43:29.402458 |                                      |    WARNING | Stopped service tripleo_keystone
2024-10-11 11:43:29.402859 |                                      |    WARNING | Stopped service tripleo_keystone_cron
2024-10-11 11:44:31.207074 |                                      |    WARNING | Stopped service tripleo_nova_virtlogd_wrapper
2024-10-11 11:44:31.208801 |                                      |    WARNING | Stopped service tripleo_ovn_controller
2024-10-11 11:44:31.210098 |                                      |    WARNING | Stopped service tripleo_nova_virtsecretd
2024-10-11 11:44:31.211791 |                                      |    WARNING | Stopped service tripleo_iscsid
2024-10-11 11:44:31.212445 |                                      |    WARNING | Stopped service tripleo_nova_virtnodedevd
2024-10-11 11:44:31.214456 |                                      |    WARNING | Stopped service tripleo_nova_virtstoraged
2024-10-11 11:44:31.215044 |                                      |    WARNING | Stopped service tripleo_nova_virtqemud
2024-10-11 11:44:31.216242 |                                      |    WARNING | Stopped service tripleo_nova_virtproxyd
2024-10-11 11:47:32.366667 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_north_db_server
2024-10-11 11:47:32.502762 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_north_db_server
2024-10-11 11:47:42.792046 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_south_db_server
2024-10-11 11:47:43.048683 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_south_db_server
2024-10-11 11:47:53.803595 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_northd
2024-10-11 11:47:54.293897 |                                      |    WARNING | Stopped service tripleo_ovn_cluster_northd
2024-10-11 11:48:01.040551 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:48:02.578835 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:48:03.119513 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:48:10.502090 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:48:10.509531 |                                      |    WARNING | Stopped service tripleo_nova_migration_target
2024-10-11 11:48:10.510536 |                                      |    WARNING | Stopped service tripleo_ovn_metadata_agent
2024-10-11 11:48:10.513469 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:48:10.514130 |                                      |    WARNING | Stopped service tripleo_nova_migration_target
2024-10-11 11:48:10.514617 |                                      |    WARNING | Stopped service tripleo_ovn_metadata_agent
2024-10-11 11:49:57.094538 |                                      |    WARNING | Stopped service tripleo_cinder_api_cron
2024-10-11 11:49:57.096113 |                                      |    WARNING | Stopped service tripleo_heat_api_cron
2024-10-11 11:49:57.096584 |                                      |    WARNING | Stopped service tripleo_heat_api_cfn
2024-10-11 11:49:57.097023 |                                      |    WARNING | Stopped service tripleo_cinder_api
2024-10-11 11:49:57.097491 |                                      |    WARNING | Stopped service tripleo_heat_api
2024-10-11 11:49:57.097948 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:49:57.098440 |                                      |    WARNING | Stopped service tripleo_nova_conductor
2024-10-11 11:49:57.098883 |                                      |    WARNING | Stopped service tripleo_nova_vnc_proxy
2024-10-11 11:49:57.099343 |                                      |    WARNING | Stopped service tripleo_neutron_api
2024-10-11 11:49:57.099759 |                                      |    WARNING | Stopped service tripleo_cinder_scheduler
2024-10-11 11:49:57.100218 |                                      |    WARNING | Stopped service tripleo_nova_scheduler
2024-10-11 11:49:57.100696 |                                      |    WARNING | Stopped service tripleo_heat_engine
2024-10-11 11:49:57.101146 |                                      |    WARNING | Stopped service tripleo_placement_api
2024-10-11 11:49:57.101579 |                                      |    WARNING | Stopped service tripleo_glance_api_cron
2024-10-11 11:49:57.102029 |                                      |    WARNING | Stopped service tripleo_glance_api
2024-10-11 11:49:57.102562 |                                      |    WARNING | Stopped service tripleo_glance_api_internal
2024-10-11 11:49:57.102995 |                                      |    WARNING | Stopped service tripleo_nova_metadata
2024-10-11 11:49:57.103482 |                                      |    WARNING | Stopped service tripleo_nova_api
2024-10-11 11:49:57.103901 |                                      |    WARNING | Stopped service tripleo_nova_api_cron
2024-10-11 11:49:57.207366 |                                      |    WARNING | Stopped service tripleo_cinder_api_cron
2024-10-11 11:49:57.209149 |                                      |    WARNING | Stopped service tripleo_heat_api_cron
2024-10-11 11:49:57.209671 |                                      |    WARNING | Stopped service tripleo_heat_api
2024-10-11 11:49:57.210124 |                                      |    WARNING | Stopped service tripleo_cinder_api
2024-10-11 11:49:57.210565 |                                      |    WARNING | Stopped service tripleo_heat_api_cfn
2024-10-11 11:49:57.210982 |                                      |    WARNING | Stopped service tripleo_logrotate_crond
2024-10-11 11:49:57.211432 |                                      |    WARNING | Stopped service tripleo_nova_conductor
2024-10-11 11:49:57.211844 |                                      |    WARNING | Stopped service tripleo_neutron_api
2024-10-11 11:49:57.212284 |                                      |    WARNING | Stopped service tripleo_cinder_scheduler
2024-10-11 11:49:57.212698 |                                      |    WARNING | Stopped service tripleo_nova_vnc_proxy
2024-10-11 11:49:57.213136 |                                      |    WARNING | Stopped service tripleo_nova_scheduler
2024-10-11 11:49:57.213564 |                                      |    WARNING | Stopped service tripleo_heat_engine
2024-10-11 11:49:57.213980 |                                      |    WARNING | Stopped service tripleo_placement_api
2024-10-11 11:49:57.214465 |                                      |    WARNING | Stopped service tripleo_glance_api_cron
2024-10-11 11:49:57.214879 |                                      |    WARNING | Stopped service tripleo_glance_api
2024-10-11 11:49:57.215351 |                                      |    WARNING | Stopped service tripleo_glance_api_internal
2024-10-11 11:49:57.215787 |                                      |    WARNING | Stopped service tripleo_nova_api
2024-10-11 11:49:57.216244 |                                      |    WARNING | Stopped service tripleo_nova_metadata
2024-10-11 11:49:57.216728 |                                      |    WARNING | Stopped service tripleo_nova_api_cron
2024-10-11 11:52:16.252390 |                                      |    WARNING | Stopped service tripleo_nova_compute
2024-10-11 11:52:16.871009 |                                      |    WARNING | Stopped service tripleo_nova_compute

Comment 21 errata-xmlrpc 2024-11-21 09:42:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHOSP 17.1.4 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:9974

Comment 22 pweeks 2024-11-25 20:59:12 UTC

*** Bug 2316429 has been marked as a duplicate of this bug. ***