2094101 – `podman` dumping core on RHCOS 4.11 + RHEL 8.6 on `aarch64`

Bug 2094101 - `podman` dumping core on RHCOS 4.11 + RHEL 8.6 on `aarch64`

Summary: `podman` dumping core on RHCOS 4.11 + RHEL 8.6 on `aarch64`

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Containers
Sub Component:
Version:	4.12
Hardware:	All
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	4.12.0
Assignee:	Matthew Heon
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-06-06 20:42 UTC by Micah Abbott
Modified:	2023-01-17 19:50 UTC (History)
CC List:	14 users (show)
Fixed In Version:	podman-4.2.0-1.rhaos4.12.el8
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-01-17 19:49:52 UTC
Target Upstream Version:
Embargoed:
Flags:	miabbott: needinfo-

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	containers podman issues 14503	None	closed	`podman` dumping core during RHCOS 4.11 + RHEL 8.6 operation on `aarch64`	2022-08-26 16:01:55 UTC
Github	containers podman pull 13950	None	Merged	systemd socket activation: check listener	2022-08-26 16:02:03 UTC
Red Hat Product Errata	RHSA-2022:7399	None	None	None	2023-01-17 19:50:00 UTC

Description Micah Abbott 2022-06-06 20:42:18 UTC

The CoreOS team is trying to rebase RHCOS 4.11 to use RHEL 8.6 content.  During our usual build + test operation of RHCOS 4.11 on `aarch64`, it was observed that `podman` dumped core:

```
Jun  6 18:30:39.815499 podman[1414]: time="2022-06-06T18:30:39Z" level=info msg="/usr/bin/podman filtering at log level info"                                  
Jun  6 18:30:39.844412 podman[1410]: time="2022-06-06T18:30:39Z" level=info msg="/usr/bin/podman filtering at log level info"
Jun  6 18:30:40.320937 podman[1414]: time="2022-06-06T18:30:40Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"                                                                                               
Jun  6 18:30:40.321224 podman[1410]: time="2022-06-06T18:30:40Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
Jun  6 18:30:40.332304 podman[1414]: time="2022-06-06T18:30:40Z" level=info msg="Setting parallel job count to 4"                               
Jun  6 18:30:40.332500 podman[1414]: time="2022-06-06T18:30:40Z" level=info msg="Using systemd socket activation to determine API endpoint"                    
Jun  6 18:30:40.339074 podman[1414]: time="2022-06-06T18:30:40Z" level=info msg="API service listening on \"/run/user/0/podman/podman.sock\""                                                                                                                                                                                 
Jun  6 18:30:40.340034 podman[1410]: time="2022-06-06T18:30:40Z" level=info msg="Setting parallel job count to 4"
Jun  6 18:30:42.615329 podman[1686]: time="2022-06-06T18:30:42Z" level=info msg="/usr/bin/podman filtering at log level info"                   
Jun  6 18:30:42.625888 podman[1685]: time="2022-06-06T18:30:42Z" level=info msg="/usr/bin/podman filtering at log level info"                                  
Jun  6 18:30:43.394662 podman[1766]: time="2022-06-06T18:30:43Z" level=info msg="/usr/bin/podman filtering at log level info"
Jun  6 18:30:43.419905 podman[1751]: time="2022-06-06T18:30:43Z" level=info msg="/usr/bin/podman filtering at log level info"                                  
Jun  6 18:30:43.568481 podman[1751]: time="2022-06-06T18:30:43Z" level=info msg="Setting parallel job count to 4"                                   
Jun  6 18:30:43.580300 podman[1766]: time="2022-06-06T18:30:43Z" level=info msg="Setting parallel job count to 4"                                                                                                                                                                                                             
Jun  6 18:30:43.581047 podman[1766]: time="2022-06-06T18:30:43Z" level=info msg="Using systemd socket activation to determine API endpoint"
Jun  6 18:30:43.584367 podman[1766]: panic: runtime error: invalid memory address or nil pointer dereference                                    
Jun  6 18:30:43.584367 podman[1766]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xaaaae9ca647c]                                             
Jun  6 18:30:43.584367 podman[1766]: goroutine 1 [running]:                                                                                                                                                                                                                                                                   
Jun  6 18:30:43.584367 podman[1766]: panic({0xaaaaea1a9dc0, 0xaaaaeb042e60})
Jun  6 18:30:43.584367 podman[1766]:         /usr/lib/golang/src/runtime/panic.go:1147 +0x41c fp=0x400093f520 sp=0x400093f440 pc=0xaaaae8bf024c 
Jun  6 18:30:43.584367 podman[1766]: runtime.panicmem()                                                                                                                                                                                                                                                                       
Jun  6 18:30:43.584367 podman[1766]:         /usr/lib/golang/src/runtime/panic.go:221 +0x58 fp=0x400093f540 sp=0x400093f520 pc=0xaaaae8bede48
Jun  6 18:30:43.584367 podman[1766]: runtime.sigpanic()                                                                                                        
Jun  6 18:30:43.584367 podman[1766]:         /usr/lib/golang/src/runtime/signal_unix.go:735 +0x1fc fp=0x400093f580 sp=0x400093f540 pc=0xaaaae8c0935c
Jun  6 18:30:43.584367 podman[1766]: github.com/containers/podman/pkg/api/server.newServer(0x4000929ba0, 0x0, {{0x0, 0x0}, {0x0, 0x0}, 0x12a05f200, {0x0, 0x0}})                                                                                                                                                              
Jun  6 18:30:43.584367 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/api/server/server.go:89 +0x1ac fp=0x400093fa00 sp=0x400093f590 pc=0xaaaae9ca647c
Jun  6 18:30:43.584367 podman[1766]: github.com/containers/podman/pkg/api/server.NewServerWithSettings(...)                                  
Jun  6 18:30:43.584367 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/api/server/server.go:64                                                                                                                                                              
Jun  6 18:30:43.584367 podman[1766]: github.com/containers/podman/cmd/podman/system.restService(0x40008a5000, 0xaaaaeb114a80, {{0x0, 0x0}, {0x0, 0x0}, 0x12a05f200, {0x0, 0x0}})
Jun  6 18:30:43.584367 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/cmd/podman/system/service_abi.go:91 +0x578 fp=0x400093fc20 sp=0x400093fa00 pc=0xaaaae9cb0bd8
Jun  6 18:30:43.584367 podman[1766]: github.com/containers/podman/cmd/podman/system.service(0xaaaaeb072ba0, {0xaaaaeb148dc8, 0x0, 0x0})
Jun  6 18:30:43.584367 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/cmd/podman/system/service.go:102 +0x280 fp=0x400093fd10 sp=0x400093fc20 pc=0xaaaae9cb0270                                                                                                
Jun  6 18:30:43.586574 podman[1766]: github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).execute(0xaaaaeb072ba0, {0x400003c0b0, 0x0, 0x0})
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:856 +0x668 fp=0x400093fdd0 sp=0x400093fd10 pc=0xaaaae91c0c78
Jun  6 18:30:43.586574 podman[1766]: github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xaaaaeb076020)
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:974 +0x410 fp=0x400093feb0 sp=0x400093fdd0 pc=0xaaaae91c13f0
Jun  6 18:30:43.586574 podman[1766]: github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).Execute(...)
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:902
Jun  6 18:30:43.586574 podman[1766]: github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...) 
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:895                                              
Jun  6 18:30:43.586574 podman[1766]: main.Execute()
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/cmd/podman/root.go:100 +0x100 fp=0x400093ff30 sp=0x400093feb0 pc=0xaaaae9d14cb0
Jun  6 18:30:43.586574 podman[1766]: main.main()
Jun  6 18:30:43.586574 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/cmd/podman/main.go:39 +0xbc fp=0x400093ff70 sp=0x400093ff30 pc=0xaaaae9d143ec
Jun  6 18:30:43.586574 podman[1766]: runtime.main()
Jun  6 18:30:43.586574 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:255 +0x284 fp=0x400093ffd0 sp=0x400093ff70 pc=0xaaaae8bf2fb4
Jun  6 18:30:43.586574 podman[1766]: runtime.goexit()
Jun  6 18:30:43.586574 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x400093ffd0 sp=0x400093ffd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.586574 podman[1766]: goroutine 2 [force gc (idle)]:
Jun  6 18:30:43.586574 podman[1766]: runtime.gopark(0xaaaaea39fb20, 0xaaaaeb10f2e0, 0x11, 0x14, 0x1)
Jun  6 18:30:43.586574 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000056fa0 sp=0x4000056f80 pc=0xaaaae8bf340c
Jun  6 18:30:43.586574 podman[1766]: runtime.goparkunlock(...)
Jun  6 18:30:43.586574 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:372
Jun  6 18:30:43.586574 podman[1766]: runtime.forcegchelper()
Jun  6 18:30:43.586574 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:306 +0xd0 fp=0x4000056fd0 sp=0x4000056fa0 pc=0xaaaae8bf3290
Jun  6 18:30:43.586574 podman[1766]: runtime.goexit()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000056fd0 sp=0x4000056fd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.587672 podman[1766]: created by runtime.init.8
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:294 +0x30
Jun  6 18:30:43.587672 podman[1766]: goroutine 3 [GC sweep wait]:
Jun  6 18:30:43.587672 podman[1766]: runtime.gopark(0xaaaaea39fb20, 0xaaaaeb111aa0, 0xc, 0x14, 0x1)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x40000577a0 sp=0x4000057780 pc=0xaaaae8bf340c
Jun  6 18:30:43.587672 podman[1766]: runtime.goparkunlock(...)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:372
Jun  6 18:30:43.587672 podman[1766]: runtime.bgsweep()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mgcsweep.go:182 +0x150 fp=0x40000577d0 sp=0x40000577a0 pc=0xaaaae8bdc670
Jun  6 18:30:43.587672 podman[1766]: runtime.goexit()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000577d0 sp=0x40000577d0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.587672 podman[1766]: created by runtime.gcenable
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mgc.go:181 +0x78
Jun  6 18:30:43.587672 podman[1766]: goroutine 4 [GC scavenge wait]:
Jun  6 18:30:43.587672 podman[1766]: runtime.gopark(0xaaaaea39fb20, 0xaaaaeb112680, 0xd, 0x14, 0x1)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000057f70 sp=0x4000057f50 pc=0xaaaae8bf340c
Jun  6 18:30:43.587672 podman[1766]: runtime.goparkunlock(...)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:372
Jun  6 18:30:43.587672 podman[1766]: runtime.bgscavenge()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mgcscavenge.go:314 +0x2c0 fp=0x4000057fd0 sp=0x4000057f70 pc=0xaaaae8bda940
Jun  6 18:30:43.587672 podman[1766]: runtime.goexit()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000057fd0 sp=0x4000057fd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.587672 podman[1766]: created by runtime.gcenable
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mgc.go:182 +0x8c
Jun  6 18:30:43.587672 podman[1766]: goroutine 5 [finalizer wait]:
Jun  6 18:30:43.587672 podman[1766]: runtime.gopark(0xaaaaea39fb20, 0xaaaaeb148cc0, 0x10, 0x14, 0x1)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000056710 sp=0x40000566f0 pc=0xaaaae8bf340c
Jun  6 18:30:43.587672 podman[1766]: runtime.goparkunlock(...)
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:372
Jun  6 18:30:43.587672 podman[1766]: runtime.runfinq()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mfinal.go:177 +0xd8 fp=0x40000567d0 sp=0x4000056710 pc=0xaaaae8bd1398
Jun  6 18:30:43.587672 podman[1766]: runtime.goexit()
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000567d0 sp=0x40000567d0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.587672 podman[1766]: created by runtime.createfing
Jun  6 18:30:43.587672 podman[1766]:         /usr/lib/golang/src/runtime/mfinal.go:157 +0x94
Jun  6 18:30:43.587672 podman[1766]: goroutine 8 [GC worker (idle)]:
Jun  6 18:30:43.587672 podman[1766]: runtime.gopark(0xaaaaea39f9b0, 0x400037d1c0, 0x18, 0x14, 0x0)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000058750 sp=0x4000058730 pc=0xaaaae8bf340c
Jun  6 18:30:43.590239 podman[1766]: runtime.gcBgMarkWorker()
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/mgc.go:1200 +0x114 fp=0x40000587d0 sp=0x4000058750 pc=0xaaaae8bd4414
Jun  6 18:30:43.590239 podman[1766]: runtime.goexit()
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000587d0 sp=0x40000587d0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.590239 podman[1766]: created by runtime.gcBgMarkStartWorkers
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/mgc.go:1124 +0x34
Jun  6 18:30:43.590239 podman[1766]: goroutine 9 [chan receive]:
Jun  6 18:30:43.590239 podman[1766]: runtime.gopark(0xaaaaea39f930, 0x400007b558, 0xe, 0x17, 0x2)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000053ec0 sp=0x4000053ea0 pc=0xaaaae8bf340c
Jun  6 18:30:43.590239 podman[1766]: runtime.chanrecv(0x400007b500, 0x4000053fb0, 0x1)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/chan.go:576 +0x5f4 fp=0x4000053f50 sp=0x4000053ec0 pc=0xaaaae8bbf2c4
Jun  6 18:30:43.590239 podman[1766]: runtime.chanrecv2(0x400007b500, 0x4000053fb0)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/chan.go:444 +0x28 fp=0x4000053f80 sp=0x4000053f50 pc=0xaaaae8bbecb8
Jun  6 18:30:43.590239 podman[1766]: github.com/containers/podman/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0xaaaaeb114960)
Jun  6 18:30:43.590239 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/k8s.io/klog/v2/klog.go:1169 +0x70 fp=0x4000053fd0 sp=0x4000053f80 pc=0xaaaae9730370
Jun  6 18:30:43.590239 podman[1766]: runtime.goexit()
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000053fd0 sp=0x4000053fd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.590239 podman[1766]: created by github.com/containers/podman/vendor/k8s.io/klog/v2.init.0
Jun  6 18:30:43.590239 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/vendor/k8s.io/klog/v2/klog.go:420 +0x150
Jun  6 18:30:43.590239 podman[1766]: goroutine 10 [select, locked to thread]:
Jun  6 18:30:43.590239 podman[1766]: runtime.gopark(0xaaaaea39fb78, 0x0, 0x9, 0x18, 0x1)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000053610 sp=0x40000535f0 pc=0xaaaae8bf340c
Jun  6 18:30:43.590239 podman[1766]: runtime.selectgo(0x40000537a0, 0x4000053798, 0x0, 0x0, 0x2, 0x1)
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/select.go:327 +0x6ec fp=0x4000053730 sp=0x4000053610 pc=0xaaaae8c050ac
Jun  6 18:30:43.590239 podman[1766]: runtime.ensureSigM.func1()
Jun  6 18:30:43.590239 podman[1766]:         /usr/lib/golang/src/runtime/signal_unix.go:890 +0x1b4 fp=0x40000537d0 sp=0x4000053730 pc=0xaaaae8c1f744
Jun  6 18:30:43.590239 podman[1766]: runtime.goexit()
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000537d0 sp=0x40000537d0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.591184 podman[1766]: created by runtime.ensureSigM
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/signal_unix.go:873 +0x10c
Jun  6 18:30:43.591184 podman[1766]: goroutine 11 [syscall]:
Jun  6 18:30:43.591184 podman[1766]: runtime.notetsleepg(0xaaaaeb149aa0, 0xffffffffffffffff)
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/lock_futex.go:236 +0x30 fp=0x4000054780 sp=0x4000054750 pc=0xaaaae8bc41d0
Jun  6 18:30:43.591184 podman[1766]: os/signal.signal_recv()
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/sigqueue.go:169 +0x110 fp=0x40000547b0 sp=0x4000054780 pc=0xaaaae8c23840
Jun  6 18:30:43.591184 podman[1766]: os/signal.loop()
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/os/signal/signal_unix.go:24 +0x20 fp=0x40000547d0 sp=0x40000547b0 pc=0xaaaae922f070
Jun  6 18:30:43.591184 podman[1766]: runtime.goexit()
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000547d0 sp=0x40000547d0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.591184 podman[1766]: created by os/signal.Notify.func1.1
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/os/signal/signal.go:151 +0x38
Jun  6 18:30:43.591184 podman[1766]: goroutine 12 [select]:
Jun  6 18:30:43.591184 podman[1766]: runtime.gopark(0xaaaaea39fb78, 0x0, 0x9, 0x18, 0x1)
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000067d10 sp=0x4000067cf0 pc=0xaaaae8bf340c
Jun  6 18:30:43.591184 podman[1766]: runtime.selectgo(0x4000067fa8, 0x4000054e90, 0x0, 0x0, 0x2, 0x1)
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/select.go:327 +0x6ec fp=0x4000067e30 sp=0x4000067d10 pc=0xaaaae8c050ac
Jun  6 18:30:43.591184 podman[1766]: github.com/containers/podman/libpod/shutdown.Start.func1()
Jun  6 18:30:43.591184 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:47 +0x80 fp=0x4000067fd0 sp=0x4000067e30 pc=0xaaaae99aa2a0
Jun  6 18:30:43.591184 podman[1766]: runtime.goexit()
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000067fd0 sp=0x4000067fd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.591184 podman[1766]: created by github.com/containers/podman/libpod/shutdown.Start
Jun  6 18:30:43.591184 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:46 +0x134
Jun  6 18:30:43.591184 podman[1766]: goroutine 14 [sleep]:
Jun  6 18:30:43.591184 podman[1766]: runtime.gopark(0xaaaaea39fb58, 0x400007c7d0, 0x13, 0x13, 0x1)
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/proc.go:366 +0xec fp=0x4000055600 sp=0x40000555e0 pc=0xaaaae8bf340c
Jun  6 18:30:43.591184 podman[1766]: time.Sleep(0x5f5e100)
Jun  6 18:30:43.591184 podman[1766]:         /usr/lib/golang/src/runtime/time.go:193 +0x138 fp=0x4000055640 sp=0x4000055600 pc=0xaaaae8c24218
Jun  6 18:30:43.601735 podman[1766]: github.com/containers/podman/libpod.(*Runtime).libimageEvents.func2(0x40001fff20, 0xaaaaea39ad38, 0x4000929ba0)
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/libpod/runtime.go:790 +0x38 fp=0x40000557c0 sp=0x4000055640 pc=0xaaaae9a7b268
Jun  6 18:30:43.601735 podman[1766]: runtime.goexit()
Jun  6 18:30:43.601735 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x40000557c0 sp=0x40000557c0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.601735 podman[1766]: created by github.com/containers/podman/libpod.(*Runtime).libimageEvents
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/libpod/runtime.go:767 +0xdc
Jun  6 18:30:43.601735 podman[1766]: goroutine 15 [runnable]:
Jun  6 18:30:43.601735 podman[1766]: github.com/containers/podman/pkg/servicereaper.reaper(0x400007a2a0)
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/servicereaper/service.go:38 fp=0x4000054fd0 sp=0x4000054fd0 pc=0xaaaae9992cc0
Jun  6 18:30:43.601735 podman[1766]: runtime.goexit()
Jun  6 18:30:43.601735 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000054fd0 sp=0x4000054fd0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.601735 podman[1766]: created by github.com/containers/podman/pkg/servicereaper.Start
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/servicereaper/service.go:35 +0x94
Jun  6 18:30:43.601735 podman[1766]: goroutine 16 [runnable]:
Jun  6 18:30:43.601735 podman[1766]: github.com/containers/podman/pkg/domain/infra.StartWatcher.func1(0x400007a360, 0x4000929ba0)
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:374 fp=0x4000055fc0 sp=0x4000055fc0 pc=0xaaaae9bd2e30
Jun  6 18:30:43.601735 podman[1766]: runtime.goexit()
Jun  6 18:30:43.601735 podman[1766]:         /usr/lib/golang/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000055fc0 sp=0x4000055fc0 pc=0xaaaae8c26eb4
Jun  6 18:30:43.601735 podman[1766]: created by github.com/containers/podman/pkg/domain/infra.StartWatcher
Jun  6 18:30:43.601735 podman[1766]:         /builddir/build/BUILD/containers-podman-2e12f02/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:374 +0x9c
Jun  6 18:30:43.843939 systemd[1622]: Started podman-pause-22df4033.scope.
Jun  6 18:30:43.847904 systemd[1622]: Started podman-pause-e1fe66be.scope.
Jun  6 18:30:44.958107 systemd-coredump[1866]: Resource limits disable core dumping for process 1766 (podman).
Jun  6 18:30:44.974946 systemd-coredump[1866]: Process 1766 (podman) of user 1000 dumped core.
Jun  6 18:30:44.995688 systemd[1622]: Started podman-pause-ff152fc7.scope.
Jun  6 18:30:44.999221 systemd[1622]: podman.service: Main process exited, code=exited, status=134/n/a
Jun  6 18:30:44.999612 systemd[1622]: podman.service: Failed with result 'exit-code'.
```

We are pulling in `podman-4.0.2-4.rhaos4.11.el8.aarch64` as part of the RHCOS 4.11 build process in this case.

xref:  https://github.com/containers/podman/issues/14503

Comment 1 Tom Sweeney 2022-06-07 14:08:09 UTC

Assigigning to @mheon to start, but have cc'd Jindrich, Dan, and Brent in case they've thoughts.

Comment 2 Matthew Heon 2022-06-07 14:13:50 UTC

I have already identified what needs to be backported, at this point it's just a question of whether we want to backport (and, if we do, starting the zstream process)

Comment 3 Tom Sweeney 2022-06-07 14:23:35 UTC

@micah.abbott Can this wait until RHEL 8.6.0.2, coming in mid-August, or do you need this backported?  @dornelas thoughts?  Unless Micah can wait for 8.6.0.2, I think we should backport to v4.0-rhel.

Comment 4 Micah Abbott 2022-06-07 15:43:38 UTC

Since `podman` is coming from `rhaos-4.11-rhel-8` branch for RHCOS, I would like to see the backport done into that branch.

As a reminder, we aren't going to be pulling the `container-tools` content from the RHEL 8.6 module for RHCOS 4.11 + RHCOS 4.12

Comment 5 Paul Holzinger 2022-06-07 16:17:16 UTC

Note, I don't think the changes from https://github.com/containers/podman/pull/13950 do anything other than fixing the panic.
This panic only happens when we get incorrect fds from systemd. With the changes it will error out.
I suspect an issue with the systemd socket activation.

Comment 7 Micah Abbott 2022-06-14 13:10:04 UTC

We're seeing evidence of this in x86_64 too and it is disrupting our CI jobs - https://github.com/openshift/os/issues/819

Comment 8 Micah Abbott 2022-06-14 13:12:07 UTC

Bumping severity to High as we are seeing this more often on more platforms

Comment 23 errata-xmlrpc 2023-01-17 19:49:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399

Note You need to log in before you can comment on or make changes to this bug.