Bug 1875289

Description of problem:
Hi folks,

I started to play with OSP16.2 and its underlying rhel8.3 components and I observe the following problem when I use podman run --rm (without the --rm there are no issues afaict):
[root@controller-0 ~]# podman run -it --net=host --rm --user=root fc9fc508dc52 sh -c 'touch /tmp/foo'
ERRO[0000] Error forwarding signal 23 to container 6cca66626a8e524b57fcb466616761c9c14539fbb9687bd4cdf4033ebfd21475: container has already been removed 

Now interestingly sometimes I only get the one-liner error above. Sometimes I will get a slightly longer error:
[root@controller-1 ~]# podman run -it --net=host --rm --user=root fc9fc508dc52 sh -c 'touch /tmp/foo'
ERRO[0000] container not running                        
container not running
ERRO[0000] Error forwarding signal 23 to container 681401cc5f2f6e19bd3e9f6f837ce5947dff2343b0a4c873076868089e9c33ec: error sending signal to container 681401c
c5f2f6e19bd3e9f6f837ce5947dff2343b0a4c873076868089e9c33ec: `/usr/bin/runc kill 681401cc5f2f6e19bd3e9f6f837ce5947dff2343b0a4c873076868089e9c33ec 23` failed: ex
it status 1 

And sometimes I will get a full go traceback http://file.rdu.redhat.com/~mbaldess/core-dump-go-rhel83.txt


A full debug log is here:
[root@controller-0 ~]# podman --log-level debug run -it --net=host --rm --user=root fc9fc508dc52 sh -c 'touch /tmp/foo'                                       
DEBU[0000] Reading configuration file "/usr/share/containers/libpod.conf" 
DEBU[0000] Merged system config "/usr/share/containers/libpod.conf": &{{false false false false false true} 0 {   [] [] []}  docker://  runc map[crun:[/usr/bi
n/crun /usr/local/bin/crun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon
 /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd   /var/run/libpod -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman []   k8s.gcr.io/pause:3.1 /pause false false  2048 shm    false} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 

DEBU[0000] Using tmp dir /var/run/libpod                                                                                                             [40/1907]
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
WARN[0000] Error loading CNI config list file /etc/cni/net.d/87-podman-bridge.conflist: error parsing configuration list: unexpected end of JSON input 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@fc9fc508dc527204749db6472cb93c1ca08f58d371b61af5c07515338c8f658b" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@fc9fc508dc527204749db6472cb93c1ca08f58d371b61af5c07515338c8f658b" 
DEBU[0000] exporting opaque data as blob "sha256:fc9fc508dc527204749db6472cb93c1ca08f58d371b61af5c07515338c8f658b" 
DEBU[0000] Using host netmode                           
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 2 for container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@fc9fc508dc527204749db6472cb93c1ca08f58d371b61af5c07515338c
8f658b" 
DEBU[0000] exporting opaque data as blob "sha256:fc9fc508dc527204749db6472cb93c1ca08f58d371b61af5c07515338c8f658b" 
DEBU[0000] created container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" 
DEBU[0000] container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" has work directory "/var/lib/containers/storage/overlay-containers/ff0
5d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata" 
DEBU[0000] container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" has run directory "/var/run/containers/storage/overlay-containers/ff05
d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata" 
DEBU[0000] New container created "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" 
DEBU[0000] container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" has CgroupParent "machine.slice/libpod-ff05d4160dbee90ec69ec25b09763f5
ff7ed2ea013e598c564802f2c35f9ec43.scope" 
DEBU[0000] Handling terminal attach                     
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/QEJ5M5BPVPPCA5EMZ4RQPII5LO:/var/lib/containers/storage/overlay/l/W3UMZ5H4DRIAMV4
WKYT22RCZJV:/var/lib/containers/storage/overlay/l/64QU6V67AZEOTRQEZOPMRJ5U5E:/var/lib/containers/storage/overlay/l/ZPZPAZU7TJ4HDB4VKSUSHYOIEE:/var/lib/contain
ers/storage/overlay/l/UGZA6PTNLNOUTIBVEDUGACBGDL,upperdir=/var/lib/containers/storage/overlay/cdcb76ee18ed40e9ac8b6aa114bb68a35dc86c9f0ea0c1c4c59431f603b5bfc9
/diff,workdir=/var/lib/containers/storage/overlay/cdcb76ee18ed40e9ac8b6aa114bb68a35dc86c9f0ea0c1c4c59431f603b5bfc9/work,context="system_u:object_r:container_f
ile_t:s0:c220,c947" 
DEBU[0000] mounted container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" at "/var/lib/containers/storage/overlay/cdcb76ee18ed40e9ac8b6a
a114bb68a35dc86c9f0ea0c1c4c59431f603b5bfc9/merged" 
DEBU[0000] Created root filesystem for container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 at /var/lib/containers/storage/overlay/cdcb7
6ee18ed40e9ac8b6aa114bb68a35dc86c9f0ea0c1c4c59431f603b5bfc9/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 to machine.slice:libpod:ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] reading hooks from /etc/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 at /var/lib/containers/storage/overlay-containers/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 -u ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata -p /var/run/containers/storage/overlay-containers/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog -t --conmon-pidfile /var/run/containers/storage/overlay-containers/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43.scope 
DEBU[0000] Received: 109190                             
INFO[0000] Got Conmon PID as 109178                     
DEBU[0000] Created container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 in OCI runtime 
DEBU[0000] Attaching to container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] connecting to socket /var/run/libpod/socket/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/attach 
DEBU[0000] Received a resize event: {Width:158 Height:42} 
DEBU[0000] Starting container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 with command [dumb-init --single-child -- sh -c touch /tmp/foo] 
DEBU[0000] Started container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] Enabling signal proxying                     
DEBU[0000] Cleaning up container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" 
DEBU[0000] Successfully cleaned up container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] Container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 storage is already unmounted, skipping... 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
ERRO[0000] Error forwarding signal 23 to container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43: container has already been removed 

Version-Release number of selected component (if applicable):
podman-1.6.4-14.module+el8.3.0+7660+b7198318.x86_64
runc-1.0.0-64.rc10.module+el8.3.0+7660+b7198318.x86_64
conmon-2.0.15-1.module+el8.3.0+7660+b7198318.x86_64
containers-common-0.1.41-3.module+el8.3.0+7660+b7198318.x86_64
container-selinux-2.130.0-1.module+el8.3.0+7660+b7198318.noarch
containernetworking-plugins-0.8.3-4.module+el8.3.0+7660+b7198318.x86_64

Additional Info:
Note I also hit the issue with a later build in 8.3 (podman-1.6.4-20.module+el8.3.0+7842+fbbcd85c.x86_64). 

If I downgrade only the podman rpm to podman-1.6.4-16.module+el8.2.0+7659+b700d80e.x86_64 it all works correctly and I can run "for i in $(seq 100); do podman run -it --net=host --rm --user=root fc9fc508dc52 sh -c 'touch /tmp/foo' 2>&1 | tee /tmp/dump$i; done" without a single error.