Bug 2155828

Description of problem:

Under some rare conditions, 'podman rm' succeeds but leaves the container running.

When a container is being stopped (or restarted), and if the main container process is not able to handle SIGTERM, podman fallback to SIGKILL after the default 10 seconds.
During these 10 seconds, a 'podman rm' command will succeed but will cancel the SIGKILL signal, leaving the container running behind.

Version-Release number of selected component (if applicable):
podman 4.2.0-4
conman 2.1.4-1
runc 1.1.4-1

How reproducible:
easily reproducible (see next section)

Steps to Reproduce:
1. Create the container (I'm also using systemd here to automatically restart the container and make it easier to catch the 10 seconds window)
$ podman create  --name test_healthcheck --health-cmd /bin/false --health-interval 5s --health-retries 3 --health-timeout 1s --health-on-failure=stop --health-start-period 5s --stop-signal SIGTERM ubi8 sleep infinity
$ podman generate systemd test_healthcheck > /etc/systemd/system/test_healthcheck.service
$ systemctl daemon-reload

2. Start the container
$ systemctl start test_healthcheck.service
$ ps -ef |grep sleep
root        6691    6689  0 14:51 ?        00:00:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep infinity
$ pstree -ps 6691
systemd(1)───conmon(6689)───sleep(6691)

3. Try to remove the container (will only work if you are in the 10s window after the first SIGTERM)
$ podman rm test_healthcheck
test_healthcheck
$ podman ps -a
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
<nothing>

Container has been removed, but the process is still running:

$ pstree -ps 6691
systemd(1)───conmon(6689)───sleep(6691)

Stopping the associated systemd unit does not help either:
$ systemctl stop test_healthcheck.service 
Dec 22 14:54:56 rhel9 systemd[1]: Stopping Podman container-4b8219383aa5f26e9b02b7be78a09791751b07861c0a9a7ae1899667661745a0.service...
Dec 22 14:54:56 rhel9 podman[6830]: Error: no container with name or ID "4b8219383aa5f26e9b02b7be78a09791751b07861c0a9a7ae1899667661745a0" found: no such container
Dec 22 14:54:56 rhel9 systemd[1]: test_healthcheck.service: Control process exited, code=exited, status=125/n/a
Dec 22 14:56:07 rhel9 systemd[1]: test_healthcheck.service: State 'stop-sigterm' timed out. Killing.
Dec 22 14:56:07 rhel9 systemd[1]: test_healthcheck.service: Killing process 6689 (conmon) with signal SIGKILL.
Dec 22 14:56:07 rhel9 systemd[1]: test_healthcheck.service: Main process exited, code=killed, status=9/KILL
Dec 22 14:56:07 rhel9 podman[6837]: Error: no container with name or ID "4b8219383aa5f26e9b02b7be78a09791751b07861c0a9a7ae1899667661745a0" found: no such container
Dec 22 14:56:07 rhel9 systemd[1]: test_healthcheck.service: Control process exited, code=exited, status=125/n/a
Dec 22 14:56:07 rhel9 systemd[1]: test_healthcheck.service: Failed with result 'exit-code'.
Dec 22 14:56:07 rhel9 systemd[1]: Stopped Podman container-4b8219383aa5f26e9b02b7be78a09791751b07861c0a9a7ae1899667661745a0.service.


$ pstree -ps 6691
systemd(1)───sleep(6691)


Actual results:
The container is still running, and the containerized process is orphaned.

Expected results:
The container is stopped and then removed.

Additional info:
Same issue on both 8.7 with podman 4.2.0-1 and 9.2 with podman 4.3.1-3