RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1875289 - podman run with --rm errors out/segfaults in container-tools-2.0-8.3.0
Summary: podman run with --rm errors out/segfaults in container-tools-2.0-8.3.0
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: podman
Version: 8.3
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: 8.0
Assignee: Jindrich Novy
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1882267
TreeView+ depends on / blocked
 
Reported: 2020-09-03 08:28 UTC by Michele Baldessari
Modified: 2021-09-03 15:25 UTC (History)
14 users (show)

Fixed In Version: podman-1.6.4-21.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1882267 (view as bug list)
Environment:
Last Closed: 2020-11-04 03:45:29 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:4770 0 None None None 2020-11-04 03:46:09 UTC

Description Michele Baldessari 2020-09-03 08:28:04 UTC 
Description of problem:
Hi folks,

I started to play with OSP16.2 and its underlying rhel8.3 components and I observe the following problem when I use podman run --rm (without the --rm there are no issues afaict):
[root@controller-0 ~]# podman run -it --net=host --rm --user=root fc9fc508dc52 sh -c 'touch /tmp/foo'
ERRO[0000] Error forwarding signal 23 to container 6cca66626a8e524b57fcb466616761c9c14539fbb9687bd4cdf4033ebfd21475: container has already been removed 

Now interestingly sometimes I only get the one-liner error above. Sometimes I will get a slightly longer error:
[root@controller-1 ~]# podman run -it --net=host --rm --user=root fc9fc508dc52 sh -c 'touch /tmp/foo'
ERRO[0000] container not running                        
container not running
ERRO[0000] Error forwarding signal 23 to container 681401cc5f2f6e19bd3e9f6f837ce5947dff2343b0a4c873076868089e9c33ec: error sending signal to container 681401c
c5f2f6e19bd3e9f6f837ce5947dff2343b0a4c873076868089e9c33ec: `/usr/bin/runc kill 681401cc5f2f6e19bd3e9f6f837ce5947dff2343b0a4c873076868089e9c33ec 23` failed: ex
it status 1 

And sometimes I will get a full go traceback http://file.rdu.redhat.com/~mbaldess/core-dump-go-rhel83.txt


A full debug log is here:
[root@controller-0 ~]# podman --log-level debug run -it --net=host --rm --user=root fc9fc508dc52 sh -c 'touch /tmp/foo'                                       
DEBU[0000] Reading configuration file "/usr/share/containers/libpod.conf" 
DEBU[0000] Merged system config "/usr/share/containers/libpod.conf": &{{false false false false false true} 0 {   [] [] []}  docker://  runc map[crun:[/usr/bi
n/crun /usr/local/bin/crun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon
 /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd   /var/run/libpod -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman []   k8s.gcr.io/pause:3.1 /pause false false  2048 shm    false} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 

DEBU[0000] Using tmp dir /var/run/libpod                                                                                                             [40/1907]
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
WARN[0000] Error loading CNI config list file /etc/cni/net.d/87-podman-bridge.conflist: error parsing configuration list: unexpected end of JSON input 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@fc9fc508dc527204749db6472cb93c1ca08f58d371b61af5c07515338c8f658b" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@fc9fc508dc527204749db6472cb93c1ca08f58d371b61af5c07515338c8f658b" 
DEBU[0000] exporting opaque data as blob "sha256:fc9fc508dc527204749db6472cb93c1ca08f58d371b61af5c07515338c8f658b" 
DEBU[0000] Using host netmode                           
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 2 for container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@fc9fc508dc527204749db6472cb93c1ca08f58d371b61af5c07515338c
8f658b" 
DEBU[0000] exporting opaque data as blob "sha256:fc9fc508dc527204749db6472cb93c1ca08f58d371b61af5c07515338c8f658b" 
DEBU[0000] created container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" 
DEBU[0000] container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" has work directory "/var/lib/containers/storage/overlay-containers/ff0
5d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata" 
DEBU[0000] container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" has run directory "/var/run/containers/storage/overlay-containers/ff05
d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata" 
DEBU[0000] New container created "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" 
DEBU[0000] container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" has CgroupParent "machine.slice/libpod-ff05d4160dbee90ec69ec25b09763f5
ff7ed2ea013e598c564802f2c35f9ec43.scope" 
DEBU[0000] Handling terminal attach                     
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/QEJ5M5BPVPPCA5EMZ4RQPII5LO:/var/lib/containers/storage/overlay/l/W3UMZ5H4DRIAMV4
WKYT22RCZJV:/var/lib/containers/storage/overlay/l/64QU6V67AZEOTRQEZOPMRJ5U5E:/var/lib/containers/storage/overlay/l/ZPZPAZU7TJ4HDB4VKSUSHYOIEE:/var/lib/contain
ers/storage/overlay/l/UGZA6PTNLNOUTIBVEDUGACBGDL,upperdir=/var/lib/containers/storage/overlay/cdcb76ee18ed40e9ac8b6aa114bb68a35dc86c9f0ea0c1c4c59431f603b5bfc9
/diff,workdir=/var/lib/containers/storage/overlay/cdcb76ee18ed40e9ac8b6aa114bb68a35dc86c9f0ea0c1c4c59431f603b5bfc9/work,context="system_u:object_r:container_f
ile_t:s0:c220,c947" 
DEBU[0000] mounted container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" at "/var/lib/containers/storage/overlay/cdcb76ee18ed40e9ac8b6a
a114bb68a35dc86c9f0ea0c1c4c59431f603b5bfc9/merged" 
DEBU[0000] Created root filesystem for container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 at /var/lib/containers/storage/overlay/cdcb7
6ee18ed40e9ac8b6aa114bb68a35dc86c9f0ea0c1c4c59431f603b5bfc9/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 to machine.slice:libpod:ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] reading hooks from /etc/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 at /var/lib/containers/storage/overlay-containers/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 -u ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata -p /var/run/containers/storage/overlay-containers/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog -t --conmon-pidfile /var/run/containers/storage/overlay-containers/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43.scope 
DEBU[0000] Received: 109190                             
INFO[0000] Got Conmon PID as 109178                     
DEBU[0000] Created container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 in OCI runtime 
DEBU[0000] Attaching to container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] connecting to socket /var/run/libpod/socket/ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43/attach 
DEBU[0000] Received a resize event: {Width:158 Height:42} 
DEBU[0000] Starting container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 with command [dumb-init --single-child -- sh -c touch /tmp/foo] 
DEBU[0000] Started container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] Enabling signal proxying                     
DEBU[0000] Cleaning up container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43" 
DEBU[0000] Successfully cleaned up container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 
DEBU[0000] Container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43 storage is already unmounted, skipping... 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
ERRO[0000] Error forwarding signal 23 to container ff05d4160dbee90ec69ec25b09763f5ff7ed2ea013e598c564802f2c35f9ec43: container has already been removed 

Version-Release number of selected component (if applicable):
podman-1.6.4-14.module+el8.3.0+7660+b7198318.x86_64
runc-1.0.0-64.rc10.module+el8.3.0+7660+b7198318.x86_64
conmon-2.0.15-1.module+el8.3.0+7660+b7198318.x86_64
containers-common-0.1.41-3.module+el8.3.0+7660+b7198318.x86_64
container-selinux-2.130.0-1.module+el8.3.0+7660+b7198318.noarch
containernetworking-plugins-0.8.3-4.module+el8.3.0+7660+b7198318.x86_64

Additional Info:
Note I also hit the issue with a later build in 8.3 (podman-1.6.4-20.module+el8.3.0+7842+fbbcd85c.x86_64). 

If I downgrade only the podman rpm to podman-1.6.4-16.module+el8.2.0+7659+b700d80e.x86_64 it all works correctly and I can run "for i in $(seq 100); do podman run -it --net=host --rm --user=root fc9fc508dc52 sh -c 'touch /tmp/foo' 2>&1 | tee /tmp/dump$i; done" without a single error.
Comment 1 Matthew Heon 2020-09-03 13:37:00 UTC 
Ouch. It looks like Podman 1.6 in the 2.0 stream was rebuilt with a newer Golang. Unfortunately, the newer Go runtime made the decision that Signal 23 is reserved exclusively for its use (for preempting threads) and sends several of them per second. In older Podman versions, our signal-proxying logic will unconditionally forward these into the container, and can (potentially) trigger error messages like the one you saw as part of a race around container stop. We addressed the race and stopped forwarding the signal in newer Podman, but the 1.6 series never got the patch because it was only built with the older Go compiler.
Comment 4 Matthew Heon 2020-09-03 15:25:07 UTC 
Upstream patches to resolve:

https://github.com/containers/podman/commit/868ee6db7057a63e09dc67b7448a6f13efcdddd3 (Closes the race around container exit and sig-proxy)
https://github.com/containers/podman/commit/e6fba1e44898304a0c5560aaecdee53beda1034f (Ignores Signal 23 so we don't spam containers with excess signals)
Comment 13 Alex Jia 2020-09-15 08:54:00 UTC 
I can reproduce this bug on podman-1.6.4-14.module+el8.3.0+7660+b7198318 w/ runc-1.0.0-64.rc10.module+el8.3.0+7994+3dff63cb.

[root@ibm-x3650m4-01-vm-02 ~]# podman run -it --net=host --rm --user=root ecbc6f53bba0 sh -c 'touch /tmp/foo && ls /tmp'
foo  ks-script-7n1migha
ERRO[0000] Error forwarding signal 23 to container 2895faceb0870a9c1be9d694d17d3073362038943d713a1dfecbf9ebb8ff3752: container has already been removed 

And verify it on podman-1.6.4-21.module+el8.3.0+7994+3dff63cb.x86_64 w/ runc-1.0.0-64.rc10.module+el8.3.0+7994+3dff63cb

[root@ibm-x3650m4-01-vm-02 ~]# podman run -it --net=host --rm --user=root ecbc6f53bba0 sh -c 'touch /tmp/foo && ls /tmp'
foo  ks-script-7n1migha
[root@ibm-x3650m4-01-vm-02 ~]# echo $?
0
Comment 16 errata-xmlrpc 2020-11-04 03:45:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (container-tools:2.0 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4770
Note You need to log in before you can comment on or make changes to this bug.