Description of problem: Post system powercycle, all podman commands fail with error Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman socket: Get "http://d/v4.1.1/libpod/_ping": dial unix ///run/podman/podman.sock: connect: resource temporarily unavailable Version-Release number of selected component (if applicable): RHEL 8.6 podman-4.1.1-7.module+el8.6.0+16889+9f154fa9.x86_64 How reproducible: Intermittent but it got reproduced on 3 out of 4 setups Steps to Reproduce: 1. Powercycle the machine 2. Post powercycle execute any podman cli Actual results: # podman network ls Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman socket: Get "http://d/v4.1.1/libpod/_ping": dial unix ///run/podman/podman.sock: connect: resource temporarily unavailable ]# podman image list Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman socket: Get "http://d/v4.1.1/libpod/_ping": dial unix ///run/podman/podman.sock: connect: resource temporarily unavailable Expected results: # podman network ls NETWORK ID NAME DRIVER f0906bc7f3cb bond0 macvlan 2f259bab93aa podman bridge Additional info: Following errors seen in the /var/log/messages <30>1 2022-11-22T04:04:40.309+00:00 nbapp837 infra-network-control.sh 11201 - - panic: invalid freelist page: 5918356779941454958, page type is branch <30>1 2022-11-22T04:04:40.309+00:00 nbapp837 infra-network-control.sh 11201 - - goroutine 1 [running]: <30>1 2022-11-22T04:04:40.338+00:00 nbapp837 infra-network-control.sh 11201 - - panic({0x55a4df2c6460, 0xc0003c8110}) <30>1 2022-11-22T04:04:40.338+00:00 nbapp837 infra-network-control.sh 11201 - - #011/usr/lib/golang/src/runtime/panic.go:1147 +0x3a8 fp=0xc0004cfb28 sp=0xc0004cfa68 pc=0x55a4dde990e8 <30>1 2022-11-22T04:04:40.347+00:00 nbapp837 infra-network-control.sh 11201 - - github.com/containers/podman/vendor/go.etcd.io/bbolt.(*freelist).read(0x55a4ded74981, 0x7f01e453c000) <30>1 2022-11-22T04:04:40.347+00:00 nbapp837 infra-network-control.sh 11201 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/freelist.go:266 +0x234 fp=0xc0004cfbc8 sp=0xc0004cfb28 pc=0x55a4de634e94 <30>1 2022-11-22T04:04:40.347+00:00 nbapp837 infra-network-control.sh 11201 - - github.com/containers/podman/vendor/go.etcd.io/bbolt.(*DB).loadFreelist.func1() <30>1 2022-11-22T04:04:40.348+00:00 nbapp837 infra-network-control.sh 11201 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:323 +0xae fp=0xc0004cfbf8 sp=0xc0004cfbc8 pc=0x55a4de62fe0e <30>1 2022-11-22T04:04:40.348+00:00 nbapp837 infra-network-control.sh 11201 - - sync.(*Once).doSlow(0xc000d261c8, 0x10) <30>1 2022-11-22T04:04:40.348+00:00 nbapp837 infra-network-control.sh 11201 - - #011/usr/lib/golang/src/sync/once.go:68 +0xd2 fp=0xc0004cfc60 sp=0xc0004cfbf8 pc=0x55a4dded76f2 <30>1 2022-11-22T04:04:40.348+00:00 nbapp837 infra-network-control.sh 11201 - - sync.(*Once).Do(...) <30>1 2022-11-22T04:04:40.348+00:00 nbapp837 infra-network-control.sh 11201 - - #011/usr/lib/golang/src/sync/once.go:59 <30>1 2022-11-22T04:04:40.348+00:00 nbapp837 infra-network-control.sh 11201 - - github.com/containers/podman/vendor/go.etcd.io/bbolt.(*DB).loadFreelist(0xc000d26000) <30>1 2022-11-22T04:04:40.348+00:00 nbapp837 infra-network-control.sh 11201 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:316 +0x47 fp=0xc0004cfc90 sp=0xc0004cfc60 pc=0x55a4de62fd27 <30>1 2022-11-22T04:04:40.348+00:00 nbapp837 infra-network-control.sh 11201 - - github.com/containers/podman/vendor/go.etcd.io/bbolt.Open({0xc0002000f0, 0x30}, 0xdeda7550, 0x0) <30>1 2022-11-22T04:04:40.348+00:00 nbapp837 infra-network-control.sh 11201 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:293 +0x46b fp=0xc0004d0d68 sp=0xc0004cfc90 pc=0x55a4de62fa8b <30>1 2022-11-22T04:04:40.365+00:00 nbapp837 infra-network-control.sh 11201 - - github.com/containers/podman/libpod.NewBoltState({0xc0002000f0, 0x30}, 0xc000c99380) <30>1 2022-11-22T04:04:40.365+00:00 nbapp837 infra-network-control.sh 11201 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/boltdb_state.go:77 +0x152 fp=0xc0004d0f80 sp=0xc0004d0d68 pc=0x55a4dea61b92 <30>1 2022-11-22T04:04:40.365+00:00 nbapp837 infra-network-control.sh 11201 - - github.com/containers/podman/libpod.makeRuntime(0xc000c99380) <30>1 2022-11-22T04:04:40.365+00:00 nbapp837 infra-network-control.sh 11201 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:325 +0x189 fp=0xc0004d13b0 sp=0xc0004d0f80 pc=0x55a4deb28989 <30>1 2022-11-22T04:04:40.365+00:00 nbapp837 infra-network-control.sh 11201 - - github.com/containers/podman/libpod.newRuntimeFromConfig(0xc000cc7c00, {0xc0004d17b0, 0x0, 0x55a4df5115b0}) <30>1 2022-11-22T04:04:40.365+00:00 nbapp837 infra-network-control.sh 11201 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:227 +0x3d7 fp=0xc0004d16d0 sp=0xc0004d13b0 pc=0x55a4deb281d7 <30>1 2022-11-22T04:04:40.365+00:00 nbapp837 infra-network-control.sh 11201 - - github.com/containers/podman/libpod.NewRuntime({0xc000c7ca00, 0x55a4ded770cb}, {0xc0004d17b0, 0x0, 0x0}) <30>1 2022-11-22T04:04:40.365+00:00 nbapp837 infra-network-control.sh 11201 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:170 +0x51 fp=0xc0004d1700 sp=0xc0004d16d0 pc=0x55a4deb27db1 <30>1 2022-11-22T04:04:40.366+00:00 nbapp837 infra-network-control.sh 11201 - - github.com/containers/podman/pkg/domain/infra.getRuntime({0x55a4df560498, 0xc000040048}, 0xc000c7ca00, 0xc0004d1a80) <30>1 2022-11-22T04:04:40.366+00:00 nbapp837 infra-network-control.sh 11201 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:254 +0x19b7 fp=0xc0004d1a60 sp=0xc0004d1700 pc=0x55a4dec28437 <30>1 2022-11-22T04:04:40.366+00:00 nbapp837 infra-network-control.sh 11201 - - github.com/containers/podman/pkg/domain/infra.GetRuntime.func1() <30>1 2022-11-22T04:04:40.366+00:00 nbapp837 infra-network-control.sh 11201 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:80 +0x49 fp=0xc0004d1ab0 sp=0xc0004d1a60 pc=0x55a4dec26a09 <30>1 2022-11-22T20:10:23.219+00:00 nbapp817 infra-network-control.sh 192536 - - panic: invalid freelist page: 66, page type is leaf <30>1 2022-11-22T20:10:23.219+00:00 nbapp817 infra-network-control.sh 192536 - - goroutine 1 [running]: <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - panic({0x55d5c5777460, 0xc00026e920}) <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - #011/usr/lib/golang/src/runtime/panic.go:1147 +0x3a8 fp=0xc000a05b28 sp=0xc000a05a68 pc=0x55d5c434a0e8 <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - github.com/containers/podman/vendor/go.etcd.io/bbolt.(*freelist).read(0x55d5c5225981, 0x7f7f34263000) <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/freelist.go:266 +0x234 fp=0xc000a05bc8 sp=0xc000a05b28 pc=0x55d5c4ae5e94 <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - github.com/containers/podman/vendor/go.etcd.io/bbolt.(*DB).loadFreelist.func1() <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:323 +0xae fp=0xc000a05bf8 sp=0xc000a05bc8 pc=0x55d5c4ae0e0e <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - sync.(*Once).doSlow(0xc0001d7608, 0x10) <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - #011/usr/lib/golang/src/sync/once.go:68 +0xd2 fp=0xc000a05c60 sp=0xc000a05bf8 pc=0x55d5c43886f2 <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - sync.(*Once).Do(...) <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - #011/usr/lib/golang/src/sync/once.go:59 <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - github.com/containers/podman/vendor/go.etcd.io/bbolt.(*DB).loadFreelist(0xc0001d7440) <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:316 +0x47 fp=0xc000a05c90 sp=0xc000a05c60 pc=0x55d5c4ae0d27 <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - github.com/containers/podman/vendor/go.etcd.io/bbolt.Open({0xc0001c7650, 0x30}, 0xc5258550, 0x0) <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:293 +0x46b fp=0xc000a06d68 sp=0xc000a05c90 pc=0x55d5c4ae0a8b <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - github.com/containers/podman/libpod.NewBoltState({0xc0001c7650, 0x30}, 0xc000f696c0) <30>1 2022-11-22T20:10:23.220+00:00 nbapp817 infra-network-control.sh 192536 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/boltdb_state.go:77 +0x152 fp=0xc000a06f80 sp=0xc000a06d68 pc=0x55d5c4f12b92 <30>1 2022-11-22T20:10:23.221+00:00 nbapp817 infra-network-control.sh 192536 - - github.com/containers/podman/libpod.makeRuntime(0xc000f696c0) <30>1 2022-11-22T20:10:23.221+00:00 nbapp817 infra-network-control.sh 192536 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:325 +0x189 fp=0xc000a073b0 sp=0xc000a06f80 pc=0x55d5c4fd9989 <30>1 2022-11-22T20:10:23.221+00:00 nbapp817 infra-network-control.sh 192536 - - github.com/containers/podman/libpod.newRuntimeFromConfig(0xc000fa7c00, {0xc000a077b0, 0x0, 0x55d5c59c25b0}) <30>1 2022-11-22T20:10:23.221+00:00 nbapp817 infra-network-control.sh 192536 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:227 +0x3d7 fp=0xc000a076d0 sp=0xc000a073b0 pc=0x55d5c4fd91d7 <30>1 2022-11-22T20:10:23.221+00:00 nbapp817 infra-network-control.sh 192536 - - github.com/containers/podman/libpod.NewRuntime({0xc000f5ca00, 0x55d5c52280cb}, {0xc000a077b0, 0x0, 0x0}) <30>1 2022-11-22T20:10:23.221+00:00 nbapp817 infra-network-control.sh 192536 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:170 +0x51 fp=0xc000a07700 sp=0xc000a076d0 pc=0x55d5c4fd8db1 <30>1 2022-11-22T20:10:23.221+00:00 nbapp817 infra-network-control.sh 192536 - - github.com/containers/podman/pkg/domain/infra.getRuntime({0x55d5c5a11498, 0xc000040048}, 0xc000f5ca00, 0xc000a07a80) <30>1 2022-11-22T20:10:23.221+00:00 nbapp817 infra-network-control.sh 192536 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:254 +0x19b7 fp=0xc000a07a60 sp=0xc000a07700 pc=0x55d5c50d9437 <30>1 2022-11-22T20:10:23.221+00:00 nbapp817 infra-network-control.sh 192536 - - github.com/containers/podman/pkg/domain/infra.GetRuntime.func1() <30>1 2022-11-22T20:10:23.221+00:00 nbapp817 infra-network-control.sh 192536 - - #011/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:80 +0x49 fp=0xc000a07ab0 sp=0xc000a07a60 pc=0x55d5c50d7a09 On two setups we see the following error: <30>1 2022-11-22T03:58:14.521+00:00 nbapp836 infra-network-control.sh 10637 - - Error: container 3ef225efde16ba8c3395134ca80ae8908b9d57de38a42e450b1363e7e20c2438 missing state in DB: internal libpod error <30>1 2022-11-22T03:58:14.522+00:00 nbapp836 podman 12681 - - Error: container 3ef225efde16ba8c3395134ca80ae8908b9d57de38a42e450b1363e7e20c2438 missing state in DB: internal libpod error <29>1 2022-11-22T03:58:14.527+00:00 nbapp836 systemd 1 - - podman.service: Main process exited, code=exited, status=125/n/a <28>1 2022-11-22T03:58:14.527+00:00 nbapp836 systemd 1 - - podman.service: Failed with result 'exit-code'.
This is an critical issue and needs to be looked on urgent basis. Also, please let us know if there is any workaround exists for this issue.
would it be possible to attach the DB in question?
also when trying these commands that fail with connection errors, is the system service running? are there logs? The message suggests absolutely not ... does `podman --log-level=debug system service -t0` reveal anything?
did you get a core?
Created attachment 1928389 [details] Bolt DB
[root@nbapp817 ~]# systemctl status podman ● podman.service - Podman API Service Loaded: loaded (/usr/lib/systemd/system/podman.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2022-11-29 18:21:23 UTC; 1ms ago Docs: man:podman-system-service(1) Main PID: 3933163 (podman) Tasks: 1 (limit: 3355442) Memory: 1.8M CGroup: /system.slice/podman.service └─3933163 /usr/bin/podman --log-level=info system service Nov 29 18:21:23 nbapp817 systemd[1]: Starting Podman API Service... Nov 29 18:21:23 nbapp817 systemd[1]: Started Podman API Service. [root@nbapp817 ~]# systemctl status podman.socket ● podman.socket - Podman API Socket Loaded: loaded (/usr/lib/systemd/system/podman.socket; enabled; vendor preset: disabled) Active: active (running) since Tue 2022-11-22 20:10:20 UTC; 6 days ago Docs: man:podman-system-service(1) Listen: /run/podman/podman.sock (Stream) Tasks: 0 (limit: 3355442) Memory: 0B CGroup: /system.slice/podman.socket Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
[root@nbapp817 ~]# podman --log-level=debug system service -t0 INFO[0000] podman filtering at log level debug DEBU[0000] Called service.PersistentPreRunE(podman --log-level=debug system service -t0) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Merged system config "/etc/containers/containers.conf.d/containers.conf" DEBU[0000] environment variable TMPDIR is already defined, skip the settings from containers.conf DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db panic: invalid freelist page: 66, page type is leaf goroutine 1 [running]: panic({0x55e960504460, 0xc000503d80}) /usr/lib/golang/src/runtime/panic.go:1147 +0x3a8 fp=0xc000595b28 sp=0xc000595a68 pc=0x55e95f0d70e8 github.com/containers/podman/vendor/go.etcd.io/bbolt.(*freelist).read(0x55e95ffb2981, 0x7fcbc42a8000) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/freelist.go:266 +0x234 fp=0xc000595bc8 sp=0xc000595b28 pc=0x55e95f872e94 github.com/containers/podman/vendor/go.etcd.io/bbolt.(*DB).loadFreelist.func1() /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:323 +0xae fp=0xc000595bf8 sp=0xc000595bc8 pc=0x55e95f86de0e sync.(*Once).doSlow(0xc000625f08, 0x10) /usr/lib/golang/src/sync/once.go:68 +0xd2 fp=0xc000595c60 sp=0xc000595bf8 pc=0x55e95f1156f2 sync.(*Once).Do(...) /usr/lib/golang/src/sync/once.go:59 github.com/containers/podman/vendor/go.etcd.io/bbolt.(*DB).loadFreelist(0xc000625d40) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:316 +0x47 fp=0xc000595c90 sp=0xc000595c60 pc=0x55e95f86dd27 github.com/containers/podman/vendor/go.etcd.io/bbolt.Open({0xc00020b740, 0x30}, 0x5ffe5550, 0x0) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:293 +0x46b fp=0xc000596d68 sp=0xc000595c90 pc=0x55e95f86da8b github.com/containers/podman/libpod.NewBoltState({0xc00020b740, 0x30}, 0xc000da1380) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/boltdb_state.go:77 +0x152 fp=0xc000596f80 sp=0xc000596d68 pc=0x55e95fc9fb92 github.com/containers/podman/libpod.makeRuntime(0xc000da1380) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:325 +0x189 fp=0xc0005973b0 sp=0xc000596f80 pc=0x55e95fd66989 github.com/containers/podman/libpod.newRuntimeFromConfig(0xc000dcdc00, {0xc0005977b0, 0x0, 0x55e96074f5b0}) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:227 +0x3d7 fp=0xc0005976d0 sp=0xc0005973b0 pc=0x55e95fd661d7 github.com/containers/podman/libpod.NewRuntime({0xc0007fda00, 0x55e95ffb50cb}, {0xc0005977b0, 0x0, 0x0}) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:170 +0x51 fp=0xc000597700 sp=0xc0005976d0 pc=0x55e95fd65db1 github.com/containers/podman/pkg/domain/infra.getRuntime({0x55e96079e498, 0xc0000420f8}, 0xc0007fda00, 0xc000597a80) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:254 +0x19b7 fp=0xc000597a60 sp=0xc000597700 pc=0x55e95fe66437 github.com/containers/podman/pkg/domain/infra.GetRuntime.func1() /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:80 +0x49 fp=0xc000597ab0 sp=0xc000597a60 pc=0x55e95fe64a09 sync.(*Once).doSlow(0x1b0000000000001e, 0x1e) /usr/lib/golang/src/sync/once.go:68 +0xd2 fp=0xc000597b18 sp=0xc000597ab0 pc=0x55e95f1156f2 sync.(*Once).Do(...) /usr/lib/golang/src/sync/once.go:59 github.com/containers/podman/pkg/domain/infra.GetRuntime({0x55e96079e498, 0xc0000420f8}, 0xc000dc87e0, 0x55e95ffbbdab) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:79 +0x78 fp=0xc000597b60 sp=0xc000597b18 pc=0x55e95fe64958 github.com/containers/podman/pkg/domain/infra.NewLibpodImageRuntime(0x55e961277600, 0xc000597bd0) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_proxy.go:25 +0x34 fp=0xc000597b98 sp=0xc000597b60 pc=0x55e95fe67fb4 github.com/containers/podman/pkg/domain/infra.NewImageEngine(0x55e95ffd2857) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_abi.go:34 +0x4a fp=0xc000597c08 sp=0xc000597b98 pc=0x55e95fe642ca github.com/containers/podman/cmd/podman/registry.NewImageEngine(0x55e96056a140, {0xc000dc87e0, 0x55e95ffbb261, 0xa}) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/cmd/podman/registry/registry.go:53 +0x51 fp=0xc000597c20 sp=0xc000597c08 pc=0x55e95fe686f1 main.persistentPreRunE(0x55e961273a00, {0xc0005023e0, 0x0, 0x1}) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/cmd/podman/root.go:216 +0x7e7 fp=0xc000597d70 sp=0xc000597c20 pc=0x55e95ffa0627 github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).execute(0x55e961273a00, {0xc00003c0e0, 0x1, 0x1}) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:835 +0x52f fp=0xc000597e30 sp=0xc000597d70 pc=0x55e95f6e626f github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x55e961277600) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:974 +0x3bc fp=0xc000597ee8 sp=0xc000597e30 pc=0x55e95f6e6a3c github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).Execute(...) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:902 github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...) /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:895 main.Execute() /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/cmd/podman/root.go:100 +0xbe fp=0xc000597f50 sp=0xc000597ee8 pc=0x55e95ff9fd1e main.main() /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/cmd/podman/main.go:40 +0x7c fp=0xc000597f80 sp=0xc000597f50 pc=0x55e95ff9f33c runtime.main() /usr/lib/golang/src/runtime/proc.go:255 +0x227 fp=0xc000597fe0 sp=0xc000597f80 pc=0x55e95f0d9c47 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000597fe8 sp=0xc000597fe0 pc=0x55e95f10b3c1 goroutine 2 [force gc (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000122fb0 sp=0xc000122f90 pc=0x55e95f0da016 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.forcegchelper() /usr/lib/golang/src/runtime/proc.go:306 +0xad fp=0xc000122fe0 sp=0xc000122fb0 pc=0x55e95f0d9ead runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000122fe8 sp=0xc000122fe0 pc=0x55e95f10b3c1 created by runtime.init.8 /usr/lib/golang/src/runtime/proc.go:294 +0x25 goroutine 3 [GC sweep wait]: runtime.gopark(0x1, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc0001237b0 sp=0xc000123790 pc=0x55e95f0da016 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.bgsweep() /usr/lib/golang/src/runtime/mgcsweep.go:182 +0xd8 fp=0xc0001237e0 sp=0xc0001237b0 pc=0x55e95f0c47b8 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001237e8 sp=0xc0001237e0 pc=0x55e95f10b3c1 created by runtime.gcenable /usr/lib/golang/src/runtime/mgc.go:181 +0x55 goroutine 4 [GC scavenge wait]: runtime.gopark(0xc000123fb8, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000123f80 sp=0xc000123f60 pc=0x55e95f0da016 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.bgscavenge() /usr/lib/golang/src/runtime/mgcscavenge.go:314 +0x288 fp=0xc000123fe0 sp=0xc000123f80 pc=0x55e95f0c2a28 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000123fe8 sp=0xc000123fe0 pc=0x55e95f10b3c1 created by runtime.gcenable /usr/lib/golang/src/runtime/mgc.go:182 +0x65 goroutine 5 [finalizer wait]: runtime.gopark(0x55e96130e760, 0xc000122770, 0xf1, 0x6e, 0x55e9612942c0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000122630 sp=0xc000122610 pc=0x55e95f0da016 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.runfinq() /usr/lib/golang/src/runtime/mfinal.go:177 +0xb3 fp=0xc0001227e0 sp=0xc000122630 pc=0x55e95f0ba293 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001227e8 sp=0xc0001227e0 pc=0x55e95f10b3c1 created by runtime.createfing /usr/lib/golang/src/runtime/mfinal.go:157 +0x45 goroutine 6 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000124760 sp=0xc000124740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001247e0 sp=0xc000124760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001247e8 sp=0xc0001247e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 7 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000124f60 sp=0xc000124f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000124fe0 sp=0xc000124f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000124fe8 sp=0xc000124fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 8 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000125760 sp=0xc000125740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001257e0 sp=0xc000125760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001257e8 sp=0xc0001257e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 9 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000125f60 sp=0xc000125f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000125fe0 sp=0xc000125f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000125fe8 sp=0xc000125fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 10 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00011e760 sp=0xc00011e740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00011e7e0 sp=0xc00011e760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00011e7e8 sp=0xc00011e7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 11 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00011ef60 sp=0xc00011ef40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00011efe0 sp=0xc00011ef60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00011efe8 sp=0xc00011efe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 12 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00011f760 sp=0xc00011f740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00011f7e0 sp=0xc00011f760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00011f7e8 sp=0xc00011f7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 18 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000194760 sp=0xc000194740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001947e0 sp=0xc000194760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001947e8 sp=0xc0001947e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 34 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000190760 sp=0xc000190740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001907e0 sp=0xc000190760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001907e8 sp=0xc0001907e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 19 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000194f60 sp=0xc000194f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000194fe0 sp=0xc000194f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000194fe8 sp=0xc000194fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 35 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000190f60 sp=0xc000190f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000190fe0 sp=0xc000190f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000190fe8 sp=0xc000190fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 13 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00011ff60 sp=0xc00011ff40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00011ffe0 sp=0xc00011ff60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00011ffe8 sp=0xc00011ffe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 36 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000191760 sp=0xc000191740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001917e0 sp=0xc000191760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001917e8 sp=0xc0001917e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 50 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000606760 sp=0xc000606740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006067e0 sp=0xc000606760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006067e8 sp=0xc0006067e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 37 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000191f60 sp=0xc000191f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000191fe0 sp=0xc000191f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000191fe8 sp=0xc000191fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 14 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000120760 sp=0xc000120740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001207e0 sp=0xc000120760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001207e8 sp=0xc0001207e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 38 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000192760 sp=0xc000192740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001927e0 sp=0xc000192760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001927e8 sp=0xc0001927e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 39 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000192f60 sp=0xc000192f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000192fe0 sp=0xc000192f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000192fe8 sp=0xc000192fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 15 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000120f60 sp=0xc000120f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000120fe0 sp=0xc000120f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000120fe8 sp=0xc000120fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 51 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000606f60 sp=0xc000606f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000606fe0 sp=0xc000606f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000606fe8 sp=0xc000606fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 40 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000193760 sp=0xc000193740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001937e0 sp=0xc000193760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001937e8 sp=0xc0001937e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 16 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000121760 sp=0xc000121740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001217e0 sp=0xc000121760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001217e8 sp=0xc0001217e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 52 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000607760 sp=0xc000607740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006077e0 sp=0xc000607760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006077e8 sp=0xc0006077e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 66 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000121f60 sp=0xc000121f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000121fe0 sp=0xc000121f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000121fe8 sp=0xc000121fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 67 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000602760 sp=0xc000602740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006027e0 sp=0xc000602760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006027e8 sp=0xc0006027e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 68 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000602f60 sp=0xc000602f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000602fe0 sp=0xc000602f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000602fe8 sp=0xc000602fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 69 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000603760 sp=0xc000603740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006037e0 sp=0xc000603760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006037e8 sp=0xc0006037e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 70 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000603f60 sp=0xc000603f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000603fe0 sp=0xc000603f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000603fe8 sp=0xc000603fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 71 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000604760 sp=0xc000604740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006047e0 sp=0xc000604760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006047e8 sp=0xc0006047e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 72 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000604f60 sp=0xc000604f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000604fe0 sp=0xc000604f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000604fe8 sp=0xc000604fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 41 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000193f60 sp=0xc000193f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000193fe0 sp=0xc000193f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000193fe8 sp=0xc000193fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 53 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000607f60 sp=0xc000607f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000607fe0 sp=0xc000607f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000607fe8 sp=0xc000607fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 42 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058a760 sp=0xc00058a740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058a7e0 sp=0xc00058a760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058a7e8 sp=0xc00058a7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 54 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000608760 sp=0xc000608740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006087e0 sp=0xc000608760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006087e8 sp=0xc0006087e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 73 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000605760 sp=0xc000605740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006057e0 sp=0xc000605760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006057e8 sp=0xc0006057e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 74 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000605f60 sp=0xc000605f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000605fe0 sp=0xc000605f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000605fe8 sp=0xc000605fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 75 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000586760 sp=0xc000586740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005867e0 sp=0xc000586760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005867e8 sp=0xc0005867e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 76 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000586f60 sp=0xc000586f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000586fe0 sp=0xc000586f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000586fe8 sp=0xc000586fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 77 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000587760 sp=0xc000587740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005877e0 sp=0xc000587760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005877e8 sp=0xc0005877e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 78 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000587f60 sp=0xc000587f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000587fe0 sp=0xc000587f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000587fe8 sp=0xc000587fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 79 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000588760 sp=0xc000588740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005887e0 sp=0xc000588760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005887e8 sp=0xc0005887e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 80 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000588f60 sp=0xc000588f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000588fe0 sp=0xc000588f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000588fe8 sp=0xc000588fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 55 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000608f60 sp=0xc000608f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000608fe0 sp=0xc000608f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000608fe8 sp=0xc000608fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 81 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000589760 sp=0xc000589740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005897e0 sp=0xc000589760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005897e8 sp=0xc0005897e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 43 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058af60 sp=0xc00058af40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058afe0 sp=0xc00058af60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058afe8 sp=0xc00058afe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 82 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000589f60 sp=0xc000589f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000589fe0 sp=0xc000589f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000589fe8 sp=0xc000589fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 44 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058b760 sp=0xc00058b740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058b7e0 sp=0xc00058b760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058b7e8 sp=0xc00058b7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 56 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000609760 sp=0xc000609740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006097e0 sp=0xc000609760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006097e8 sp=0xc0006097e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 20 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000195760 sp=0xc000195740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001957e0 sp=0xc000195760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001957e8 sp=0xc0001957e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 45 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058bf60 sp=0xc00058bf40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058bfe0 sp=0xc00058bf60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058bfe8 sp=0xc00058bfe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 57 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000609f60 sp=0xc000609f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000609fe0 sp=0xc000609f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000609fe8 sp=0xc000609fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 46 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058c760 sp=0xc00058c740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058c7e0 sp=0xc00058c760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058c7e8 sp=0xc00058c7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 58 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000610760 sp=0xc000610740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006107e0 sp=0xc000610760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006107e8 sp=0xc0006107e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 47 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058cf60 sp=0xc00058cf40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058cfe0 sp=0xc00058cf60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058cfe8 sp=0xc00058cfe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 59 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000610f60 sp=0xc000610f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000610fe0 sp=0xc000610f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000610fe8 sp=0xc000610fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 48 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058d760 sp=0xc00058d740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058d7e0 sp=0xc00058d760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058d7e8 sp=0xc00058d7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 98 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060c760 sp=0xc00060c740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060c7e0 sp=0xc00060c760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060c7e8 sp=0xc00060c7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 21 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000195f60 sp=0xc000195f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000195fe0 sp=0xc000195f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000195fe8 sp=0xc000195fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 22 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000196760 sp=0xc000196740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001967e0 sp=0xc000196760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001967e8 sp=0xc0001967e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 23 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000196f60 sp=0xc000196f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000196fe0 sp=0xc000196f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000196fe8 sp=0xc000196fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 83 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000510760 sp=0xc000510740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005107e0 sp=0xc000510760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005107e8 sp=0xc0005107e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 24 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000197760 sp=0xc000197740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001977e0 sp=0xc000197760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001977e8 sp=0xc0001977e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 25 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000197f60 sp=0xc000197f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000197fe0 sp=0xc000197f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000197fe8 sp=0xc000197fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 99 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060cf60 sp=0xc00060cf40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060cfe0 sp=0xc00060cf60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060cfe8 sp=0xc00060cfe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 100 [GC worker (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060d760 sp=0xc00060d740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060d7e0 sp=0xc00060d760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060d7e8 sp=0xc00060d7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 84 [GC worker (idle)]: runtime.gopark(0x220a44da20856, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000510f60 sp=0xc000510f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000510fe0 sp=0xc000510f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000510fe8 sp=0xc000510fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 26 [GC worker (idle)]: runtime.gopark(0x55e960751368, 0xc0004dc9a0, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050c760 sp=0xc00050c740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050c7e0 sp=0xc00050c760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050c7e8 sp=0xc00050c7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 27 [GC worker (idle)]: runtime.gopark(0x55e960751368, 0xc000682000, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050cf60 sp=0xc00050cf40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050cfe0 sp=0xc00050cf60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050cfe8 sp=0xc00050cfe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 28 [GC worker (idle)]: runtime.gopark(0x55e960751368, 0xc000682020, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050d760 sp=0xc00050d740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050d7e0 sp=0xc00050d760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050d7e8 sp=0xc00050d7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 29 [GC worker (idle)]: runtime.gopark(0x55e960751368, 0xc0004dc9c0, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050df60 sp=0xc00050df40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050dfe0 sp=0xc00050df60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050dfe8 sp=0xc00050dfe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 85 [GC worker (idle)]: runtime.gopark(0x55e960751368, 0xc000682040, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000511760 sp=0xc000511740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005117e0 sp=0xc000511760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005117e8 sp=0xc0005117e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 86 [GC worker (idle)]: runtime.gopark(0x220a44da2168f, 0xc00018e100, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000511f60 sp=0xc000511f40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000511fe0 sp=0xc000511f60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000511fe8 sp=0xc000511fe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 30 [GC worker (idle)]: runtime.gopark(0x55e960751368, 0xc000682060, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050e760 sp=0xc00050e740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050e7e0 sp=0xc00050e760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050e7e8 sp=0xc00050e7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 87 [GC worker (idle)]: runtime.gopark(0x55e960751368, 0xc000682080, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000512760 sp=0xc000512740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005127e0 sp=0xc000512760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005127e8 sp=0xc0005127e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 101 [GC worker (idle)]: runtime.gopark(0x220a44da1fe81, 0xc0006820a0, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060df60 sp=0xc00060df40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060dfe0 sp=0xc00060df60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060dfe8 sp=0xc00060dfe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 102 [GC worker (idle)]: runtime.gopark(0x55e960751368, 0xc0006820c0, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060e760 sp=0xc00060e740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060e7e0 sp=0xc00060e760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060e7e8 sp=0xc00060e7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 103 [GC worker (idle)]: runtime.gopark(0x220a44da2176b, 0xc0006820e0, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060ef60 sp=0xc00060ef40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060efe0 sp=0xc00060ef60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060efe8 sp=0xc00060efe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 104 [GC worker (idle)]: runtime.gopark(0x220a44da21776, 0xc000682100, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060f760 sp=0xc00060f740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060f7e0 sp=0xc00060f760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060f7e8 sp=0xc00060f7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 31 [GC worker (idle)]: runtime.gopark(0x55e960751368, 0xc00018e120, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050ef60 sp=0xc00050ef40 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050efe0 sp=0xc00050ef60 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050efe8 sp=0xc00050efe0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 32 [GC worker (idle)]: runtime.gopark(0x55e960751368, 0xc00018e140, 0x18, 0x14, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050f760 sp=0xc00050f740 pc=0x55e95f0da016 runtime.gcBgMarkWorker() /usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050f7e0 sp=0xc00050f760 pc=0x55e95f0bcce5 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050f7e8 sp=0xc00050f7e0 pc=0x55e95f10b3c1 created by runtime.gcBgMarkStartWorkers /usr/lib/golang/src/runtime/mgc.go:1124 +0x25 goroutine 88 [select, locked to thread]: runtime.gopark(0xc00050ffa8, 0x2, 0x90, 0xfe, 0xc00050ffa4) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000133e10 sp=0xc000133df0 pc=0x55e95f0da016 runtime.selectgo(0xc000133fa8, 0xc00050ffa0, 0x0, 0x0, 0x0, 0x1) /usr/lib/golang/src/runtime/select.go:327 +0x772 fp=0xc000133f58 sp=0xc000133e10 pc=0x55e95f0e9e12 runtime.ensureSigM.func1() /usr/lib/golang/src/runtime/signal_unix.go:890 +0x1a9 fp=0xc000133fe0 sp=0xc000133f58 pc=0x55e95f103ca9 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000133fe8 sp=0xc000133fe0 pc=0x55e95f10b3c1 created by runtime.ensureSigM /usr/lib/golang/src/runtime/signal_unix.go:873 +0xbf goroutine 114 [syscall]: runtime.notetsleepg(0x0, 0x0) /usr/lib/golang/src/runtime/lock_futex.go:236 +0x34 fp=0xc000e05fa0 sp=0xc000e05f68 pc=0x55e95f0ad6d4 os/signal.signal_recv() /usr/lib/golang/src/runtime/sigqueue.go:169 +0x98 fp=0xc000e05fc0 sp=0xc000e05fa0 pc=0x55e95f1078d8 os/signal.loop() /usr/lib/golang/src/os/signal/signal_unix.go:24 +0x19 fp=0xc000e05fe0 sp=0xc000e05fc0 pc=0x55e95f3ff919 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000e05fe8 sp=0xc000e05fe0 pc=0x55e95f10b3c1 created by os/signal.Notify.func1.1 /usr/lib/golang/src/os/signal/signal.go:151 +0x2c goroutine 89 [select]: runtime.gopark(0xc0006117b0, 0x2, 0x0, 0x0, 0xc0006116a4) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000e0fd08 sp=0xc000e0fce8 pc=0x55e95f0da016 runtime.selectgo(0xc000e0ffb0, 0xc0006116a0, 0x0, 0x0, 0x0, 0x1) /usr/lib/golang/src/runtime/select.go:327 +0x772 fp=0xc000e0fe50 sp=0xc000e0fd08 pc=0x55e95f0e9e12 github.com/containers/podman/libpod/shutdown.Start.func1() /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:47 +0x99 fp=0xc000e0ffe0 sp=0xc000e0fe50 pc=0x55e95fc80c59 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000e0ffe8 sp=0xc000e0ffe0 pc=0x55e95f10b3c1 created by github.com/containers/podman/libpod/shutdown.Start /builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:46 +0xea Aborted (core dumped) [root@nbapp817 ~]#
Created attachment 1928471 [details] core file
Hi @bbaude Please let me know if you need anything else.
We had another issue of podman panictoday after a system power cycle. We deleted /var/lib/containers/storage/libpod/bolt_state.db Removed all the stale conainers reported by 'podman ps --all --external' with Status as 'Storage' But podman image list still complained, about 'layer not known' # podman image ls Error: error retrieving label for image "b3a567473e54f9cdc3c0d8e45904e421c7c149e2cba5fe69a3327da17614b765": you may need to remove the image to resolve the error: layer not known After we removed all the images that were in error, 'podman image ls' reported empty # podman image ls REPOSITORY TAG IMAGE ID CREATED SIZE We see all layers present in /var/lib/containers/storage/overlay-layers/ We need a solution to fix this problem. Is there a way to reconstruct the image list.
It sounds like we're starting to diverge here: We have the Podman database corruption issue we were originally discussing, but now we have another case of DB corruption, this time in c/storage. Can we keep the Podman database corruption issue (bolt_state.db) here, and open a fresh issue for the similar but separate c/storage corruption (layers.json). Can you also attach a corrupted layers.json (and any other files in the `/var/lib/containers/storage/overlay-layers/` directory that end in `.json`)? Finally: are we certain there are no container creation operations going on during steady-state? The layers file should only be modified when containers are created/deleted or images are pulled.
(In reply to Matthew Heon from comment #22) > It sounds like we're starting to diverge here: We have the Podman database > corruption issue we were originally discussing, but now we have another case > of DB corruption, this time in c/storage. Can we keep the Podman database > corruption issue (bolt_state.db) here, and open a fresh issue for the > similar but separate c/storage corruption (layers.json). Can you also attach > a corrupted layers.json (and any other files in the > `/var/lib/containers/storage/overlay-layers/` directory that end in `.json`)? > > Finally: are we certain there are no container creation operations going on > during steady-state? The layers file should only be modified when containers > are created/deleted or images are pulled. Hello Matthew, We have a new case opened for the DB corruption in c/storage and we have the layers.json file available on the case as well. Since the current issue is different from the new corruption do you want me to open up a new Bug ? Regards, Carroline
For opening a new bug: Yes, please do
Mangesh, We have recently added support for sqlite into Podman in large part because of this bug. Unfortunately, we are unable to replicate this bug like you can (with boltdb); would you be willing to try a Podman where sqlite is used and see if you have better results? We are also adding a new feature which should also help this condition but it is not complete yet.