Bug 2149112
| Summary: | Podman panic post power cycle | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | mangesh.panche | ||||||
| Component: | podman | Assignee: | Matthew Heon <mheon> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | atomic-bugs <atomic-bugs> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | 8.6 | CC: | bbaude, cpippin, dwalsh, jligon, jnovy, kcleveng, lsm5, mboddu, mheon, pthomas, snangare, tsweeney, umohnani, vikas.goel | ||||||
| Target Milestone: | rc | Keywords: | DeliveryBlocker | ||||||
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
||||||
| Hardware: | x86_64 | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2023-05-10 15:19:55 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
mangesh.panche
2022-11-28 21:13:48 UTC
This is an critical issue and needs to be looked on urgent basis. Also, please let us know if there is any workaround exists for this issue. would it be possible to attach the DB in question? also when trying these commands that fail with connection errors, is the system service running? are there logs? The message suggests absolutely not ... does `podman --log-level=debug system service -t0` reveal anything? did you get a core? Created attachment 1928389 [details]
Bolt DB
[root@nbapp817 ~]# systemctl status podman
● podman.service - Podman API Service
Loaded: loaded (/usr/lib/systemd/system/podman.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2022-11-29 18:21:23 UTC; 1ms ago
Docs: man:podman-system-service(1)
Main PID: 3933163 (podman)
Tasks: 1 (limit: 3355442)
Memory: 1.8M
CGroup: /system.slice/podman.service
└─3933163 /usr/bin/podman --log-level=info system service
Nov 29 18:21:23 nbapp817 systemd[1]: Starting Podman API Service...
Nov 29 18:21:23 nbapp817 systemd[1]: Started Podman API Service.
[root@nbapp817 ~]# systemctl status podman.socket
● podman.socket - Podman API Socket
Loaded: loaded (/usr/lib/systemd/system/podman.socket; enabled; vendor preset: disabled)
Active: active (running) since Tue 2022-11-22 20:10:20 UTC; 6 days ago
Docs: man:podman-system-service(1)
Listen: /run/podman/podman.sock (Stream)
Tasks: 0 (limit: 3355442)
Memory: 0B
CGroup: /system.slice/podman.socket
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
[root@nbapp817 ~]# podman --log-level=debug system service -t0
INFO[0000] podman filtering at log level debug
DEBU[0000] Called service.PersistentPreRunE(podman --log-level=debug system service -t0)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/etc/containers/containers.conf.d/containers.conf"
DEBU[0000] environment variable TMPDIR is already defined, skip the settings from containers.conf
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
panic: invalid freelist page: 66, page type is leaf
goroutine 1 [running]:
panic({0x55e960504460, 0xc000503d80})
/usr/lib/golang/src/runtime/panic.go:1147 +0x3a8 fp=0xc000595b28 sp=0xc000595a68 pc=0x55e95f0d70e8
github.com/containers/podman/vendor/go.etcd.io/bbolt.(*freelist).read(0x55e95ffb2981, 0x7fcbc42a8000)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/freelist.go:266 +0x234 fp=0xc000595bc8 sp=0xc000595b28 pc=0x55e95f872e94
github.com/containers/podman/vendor/go.etcd.io/bbolt.(*DB).loadFreelist.func1()
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:323 +0xae fp=0xc000595bf8 sp=0xc000595bc8 pc=0x55e95f86de0e
sync.(*Once).doSlow(0xc000625f08, 0x10)
/usr/lib/golang/src/sync/once.go:68 +0xd2 fp=0xc000595c60 sp=0xc000595bf8 pc=0x55e95f1156f2
sync.(*Once).Do(...)
/usr/lib/golang/src/sync/once.go:59
github.com/containers/podman/vendor/go.etcd.io/bbolt.(*DB).loadFreelist(0xc000625d40)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:316 +0x47 fp=0xc000595c90 sp=0xc000595c60 pc=0x55e95f86dd27
github.com/containers/podman/vendor/go.etcd.io/bbolt.Open({0xc00020b740, 0x30}, 0x5ffe5550, 0x0)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:293 +0x46b fp=0xc000596d68 sp=0xc000595c90 pc=0x55e95f86da8b
github.com/containers/podman/libpod.NewBoltState({0xc00020b740, 0x30}, 0xc000da1380)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/boltdb_state.go:77 +0x152 fp=0xc000596f80 sp=0xc000596d68 pc=0x55e95fc9fb92
github.com/containers/podman/libpod.makeRuntime(0xc000da1380)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:325 +0x189 fp=0xc0005973b0 sp=0xc000596f80 pc=0x55e95fd66989
github.com/containers/podman/libpod.newRuntimeFromConfig(0xc000dcdc00, {0xc0005977b0, 0x0, 0x55e96074f5b0})
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:227 +0x3d7 fp=0xc0005976d0 sp=0xc0005973b0 pc=0x55e95fd661d7
github.com/containers/podman/libpod.NewRuntime({0xc0007fda00, 0x55e95ffb50cb}, {0xc0005977b0, 0x0, 0x0})
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/runtime.go:170 +0x51 fp=0xc000597700 sp=0xc0005976d0 pc=0x55e95fd65db1
github.com/containers/podman/pkg/domain/infra.getRuntime({0x55e96079e498, 0xc0000420f8}, 0xc0007fda00, 0xc000597a80)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:254 +0x19b7 fp=0xc000597a60 sp=0xc000597700 pc=0x55e95fe66437
github.com/containers/podman/pkg/domain/infra.GetRuntime.func1()
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:80 +0x49 fp=0xc000597ab0 sp=0xc000597a60 pc=0x55e95fe64a09
sync.(*Once).doSlow(0x1b0000000000001e, 0x1e)
/usr/lib/golang/src/sync/once.go:68 +0xd2 fp=0xc000597b18 sp=0xc000597ab0 pc=0x55e95f1156f2
sync.(*Once).Do(...)
/usr/lib/golang/src/sync/once.go:59
github.com/containers/podman/pkg/domain/infra.GetRuntime({0x55e96079e498, 0xc0000420f8}, 0xc000dc87e0, 0x55e95ffbbdab)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:79 +0x78 fp=0xc000597b60 sp=0xc000597b18 pc=0x55e95fe64958
github.com/containers/podman/pkg/domain/infra.NewLibpodImageRuntime(0x55e961277600, 0xc000597bd0)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_proxy.go:25 +0x34 fp=0xc000597b98 sp=0xc000597b60 pc=0x55e95fe67fb4
github.com/containers/podman/pkg/domain/infra.NewImageEngine(0x55e95ffd2857)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_abi.go:34 +0x4a fp=0xc000597c08 sp=0xc000597b98 pc=0x55e95fe642ca
github.com/containers/podman/cmd/podman/registry.NewImageEngine(0x55e96056a140, {0xc000dc87e0, 0x55e95ffbb261, 0xa})
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/cmd/podman/registry/registry.go:53 +0x51 fp=0xc000597c20 sp=0xc000597c08 pc=0x55e95fe686f1
main.persistentPreRunE(0x55e961273a00, {0xc0005023e0, 0x0, 0x1})
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/cmd/podman/root.go:216 +0x7e7 fp=0xc000597d70 sp=0xc000597c20 pc=0x55e95ffa0627
github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).execute(0x55e961273a00, {0xc00003c0e0, 0x1, 0x1})
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:835 +0x52f fp=0xc000597e30 sp=0xc000597d70 pc=0x55e95f6e626f
github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x55e961277600)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:974 +0x3bc fp=0xc000597ee8 sp=0xc000597e30 pc=0x55e95f6e6a3c
github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).Execute(...)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:902
github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...)
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:895
main.Execute()
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/cmd/podman/root.go:100 +0xbe fp=0xc000597f50 sp=0xc000597ee8 pc=0x55e95ff9fd1e
main.main()
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/cmd/podman/main.go:40 +0x7c fp=0xc000597f80 sp=0xc000597f50 pc=0x55e95ff9f33c
runtime.main()
/usr/lib/golang/src/runtime/proc.go:255 +0x227 fp=0xc000597fe0 sp=0xc000597f80 pc=0x55e95f0d9c47
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000597fe8 sp=0xc000597fe0 pc=0x55e95f10b3c1
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000122fb0 sp=0xc000122f90 pc=0x55e95f0da016
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.forcegchelper()
/usr/lib/golang/src/runtime/proc.go:306 +0xad fp=0xc000122fe0 sp=0xc000122fb0 pc=0x55e95f0d9ead
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000122fe8 sp=0xc000122fe0 pc=0x55e95f10b3c1
created by runtime.init.8
/usr/lib/golang/src/runtime/proc.go:294 +0x25
goroutine 3 [GC sweep wait]:
runtime.gopark(0x1, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc0001237b0 sp=0xc000123790 pc=0x55e95f0da016
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.bgsweep()
/usr/lib/golang/src/runtime/mgcsweep.go:182 +0xd8 fp=0xc0001237e0 sp=0xc0001237b0 pc=0x55e95f0c47b8
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001237e8 sp=0xc0001237e0 pc=0x55e95f10b3c1
created by runtime.gcenable
/usr/lib/golang/src/runtime/mgc.go:181 +0x55
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc000123fb8, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000123f80 sp=0xc000123f60 pc=0x55e95f0da016
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.bgscavenge()
/usr/lib/golang/src/runtime/mgcscavenge.go:314 +0x288 fp=0xc000123fe0 sp=0xc000123f80 pc=0x55e95f0c2a28
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000123fe8 sp=0xc000123fe0 pc=0x55e95f10b3c1
created by runtime.gcenable
/usr/lib/golang/src/runtime/mgc.go:182 +0x65
goroutine 5 [finalizer wait]:
runtime.gopark(0x55e96130e760, 0xc000122770, 0xf1, 0x6e, 0x55e9612942c0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000122630 sp=0xc000122610 pc=0x55e95f0da016
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.runfinq()
/usr/lib/golang/src/runtime/mfinal.go:177 +0xb3 fp=0xc0001227e0 sp=0xc000122630 pc=0x55e95f0ba293
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001227e8 sp=0xc0001227e0 pc=0x55e95f10b3c1
created by runtime.createfing
/usr/lib/golang/src/runtime/mfinal.go:157 +0x45
goroutine 6 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000124760 sp=0xc000124740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001247e0 sp=0xc000124760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001247e8 sp=0xc0001247e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 7 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000124f60 sp=0xc000124f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000124fe0 sp=0xc000124f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000124fe8 sp=0xc000124fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 8 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000125760 sp=0xc000125740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001257e0 sp=0xc000125760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001257e8 sp=0xc0001257e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 9 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000125f60 sp=0xc000125f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000125fe0 sp=0xc000125f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000125fe8 sp=0xc000125fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 10 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00011e760 sp=0xc00011e740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00011e7e0 sp=0xc00011e760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00011e7e8 sp=0xc00011e7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 11 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00011ef60 sp=0xc00011ef40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00011efe0 sp=0xc00011ef60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00011efe8 sp=0xc00011efe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 12 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00011f760 sp=0xc00011f740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00011f7e0 sp=0xc00011f760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00011f7e8 sp=0xc00011f7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 18 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000194760 sp=0xc000194740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001947e0 sp=0xc000194760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001947e8 sp=0xc0001947e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 34 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000190760 sp=0xc000190740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001907e0 sp=0xc000190760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001907e8 sp=0xc0001907e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 19 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000194f60 sp=0xc000194f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000194fe0 sp=0xc000194f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000194fe8 sp=0xc000194fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 35 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000190f60 sp=0xc000190f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000190fe0 sp=0xc000190f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000190fe8 sp=0xc000190fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 13 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00011ff60 sp=0xc00011ff40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00011ffe0 sp=0xc00011ff60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00011ffe8 sp=0xc00011ffe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 36 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000191760 sp=0xc000191740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001917e0 sp=0xc000191760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001917e8 sp=0xc0001917e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 50 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000606760 sp=0xc000606740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006067e0 sp=0xc000606760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006067e8 sp=0xc0006067e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 37 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000191f60 sp=0xc000191f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000191fe0 sp=0xc000191f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000191fe8 sp=0xc000191fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 14 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000120760 sp=0xc000120740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001207e0 sp=0xc000120760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001207e8 sp=0xc0001207e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 38 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000192760 sp=0xc000192740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001927e0 sp=0xc000192760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001927e8 sp=0xc0001927e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 39 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000192f60 sp=0xc000192f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000192fe0 sp=0xc000192f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000192fe8 sp=0xc000192fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 15 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000120f60 sp=0xc000120f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000120fe0 sp=0xc000120f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000120fe8 sp=0xc000120fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 51 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000606f60 sp=0xc000606f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000606fe0 sp=0xc000606f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000606fe8 sp=0xc000606fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 40 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000193760 sp=0xc000193740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001937e0 sp=0xc000193760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001937e8 sp=0xc0001937e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 16 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000121760 sp=0xc000121740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001217e0 sp=0xc000121760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001217e8 sp=0xc0001217e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 52 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000607760 sp=0xc000607740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006077e0 sp=0xc000607760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006077e8 sp=0xc0006077e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 66 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000121f60 sp=0xc000121f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000121fe0 sp=0xc000121f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000121fe8 sp=0xc000121fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 67 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000602760 sp=0xc000602740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006027e0 sp=0xc000602760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006027e8 sp=0xc0006027e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 68 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000602f60 sp=0xc000602f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000602fe0 sp=0xc000602f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000602fe8 sp=0xc000602fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 69 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000603760 sp=0xc000603740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006037e0 sp=0xc000603760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006037e8 sp=0xc0006037e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 70 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000603f60 sp=0xc000603f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000603fe0 sp=0xc000603f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000603fe8 sp=0xc000603fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 71 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000604760 sp=0xc000604740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006047e0 sp=0xc000604760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006047e8 sp=0xc0006047e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 72 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000604f60 sp=0xc000604f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000604fe0 sp=0xc000604f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000604fe8 sp=0xc000604fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 41 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000193f60 sp=0xc000193f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000193fe0 sp=0xc000193f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000193fe8 sp=0xc000193fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 53 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000607f60 sp=0xc000607f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000607fe0 sp=0xc000607f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000607fe8 sp=0xc000607fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 42 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058a760 sp=0xc00058a740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058a7e0 sp=0xc00058a760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058a7e8 sp=0xc00058a7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 54 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000608760 sp=0xc000608740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006087e0 sp=0xc000608760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006087e8 sp=0xc0006087e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 73 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000605760 sp=0xc000605740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006057e0 sp=0xc000605760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006057e8 sp=0xc0006057e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 74 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000605f60 sp=0xc000605f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000605fe0 sp=0xc000605f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000605fe8 sp=0xc000605fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 75 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000586760 sp=0xc000586740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005867e0 sp=0xc000586760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005867e8 sp=0xc0005867e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 76 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000586f60 sp=0xc000586f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000586fe0 sp=0xc000586f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000586fe8 sp=0xc000586fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 77 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000587760 sp=0xc000587740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005877e0 sp=0xc000587760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005877e8 sp=0xc0005877e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 78 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000587f60 sp=0xc000587f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000587fe0 sp=0xc000587f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000587fe8 sp=0xc000587fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 79 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000588760 sp=0xc000588740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005887e0 sp=0xc000588760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005887e8 sp=0xc0005887e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 80 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000588f60 sp=0xc000588f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000588fe0 sp=0xc000588f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000588fe8 sp=0xc000588fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 55 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000608f60 sp=0xc000608f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000608fe0 sp=0xc000608f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000608fe8 sp=0xc000608fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 81 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000589760 sp=0xc000589740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005897e0 sp=0xc000589760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005897e8 sp=0xc0005897e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 43 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058af60 sp=0xc00058af40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058afe0 sp=0xc00058af60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058afe8 sp=0xc00058afe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 82 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000589f60 sp=0xc000589f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000589fe0 sp=0xc000589f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000589fe8 sp=0xc000589fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 44 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058b760 sp=0xc00058b740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058b7e0 sp=0xc00058b760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058b7e8 sp=0xc00058b7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 56 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000609760 sp=0xc000609740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006097e0 sp=0xc000609760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006097e8 sp=0xc0006097e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 20 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000195760 sp=0xc000195740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001957e0 sp=0xc000195760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001957e8 sp=0xc0001957e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 45 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058bf60 sp=0xc00058bf40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058bfe0 sp=0xc00058bf60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058bfe8 sp=0xc00058bfe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 57 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000609f60 sp=0xc000609f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000609fe0 sp=0xc000609f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000609fe8 sp=0xc000609fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 46 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058c760 sp=0xc00058c740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058c7e0 sp=0xc00058c760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058c7e8 sp=0xc00058c7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 58 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000610760 sp=0xc000610740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0006107e0 sp=0xc000610760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006107e8 sp=0xc0006107e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 47 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058cf60 sp=0xc00058cf40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058cfe0 sp=0xc00058cf60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058cfe8 sp=0xc00058cfe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 59 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000610f60 sp=0xc000610f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000610fe0 sp=0xc000610f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000610fe8 sp=0xc000610fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 48 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00058d760 sp=0xc00058d740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00058d7e0 sp=0xc00058d760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00058d7e8 sp=0xc00058d7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 98 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060c760 sp=0xc00060c740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060c7e0 sp=0xc00060c760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060c7e8 sp=0xc00060c7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 21 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000195f60 sp=0xc000195f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000195fe0 sp=0xc000195f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000195fe8 sp=0xc000195fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 22 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000196760 sp=0xc000196740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001967e0 sp=0xc000196760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001967e8 sp=0xc0001967e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 23 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000196f60 sp=0xc000196f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000196fe0 sp=0xc000196f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000196fe8 sp=0xc000196fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 83 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000510760 sp=0xc000510740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005107e0 sp=0xc000510760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005107e8 sp=0xc0005107e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 24 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000197760 sp=0xc000197740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0001977e0 sp=0xc000197760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0001977e8 sp=0xc0001977e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 25 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000197f60 sp=0xc000197f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000197fe0 sp=0xc000197f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000197fe8 sp=0xc000197fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 99 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060cf60 sp=0xc00060cf40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060cfe0 sp=0xc00060cf60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060cfe8 sp=0xc00060cfe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 100 [GC worker (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060d760 sp=0xc00060d740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060d7e0 sp=0xc00060d760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060d7e8 sp=0xc00060d7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 84 [GC worker (idle)]:
runtime.gopark(0x220a44da20856, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000510f60 sp=0xc000510f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000510fe0 sp=0xc000510f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000510fe8 sp=0xc000510fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 26 [GC worker (idle)]:
runtime.gopark(0x55e960751368, 0xc0004dc9a0, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050c760 sp=0xc00050c740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050c7e0 sp=0xc00050c760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050c7e8 sp=0xc00050c7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 27 [GC worker (idle)]:
runtime.gopark(0x55e960751368, 0xc000682000, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050cf60 sp=0xc00050cf40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050cfe0 sp=0xc00050cf60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050cfe8 sp=0xc00050cfe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 28 [GC worker (idle)]:
runtime.gopark(0x55e960751368, 0xc000682020, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050d760 sp=0xc00050d740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050d7e0 sp=0xc00050d760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050d7e8 sp=0xc00050d7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 29 [GC worker (idle)]:
runtime.gopark(0x55e960751368, 0xc0004dc9c0, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050df60 sp=0xc00050df40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050dfe0 sp=0xc00050df60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050dfe8 sp=0xc00050dfe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 85 [GC worker (idle)]:
runtime.gopark(0x55e960751368, 0xc000682040, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000511760 sp=0xc000511740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005117e0 sp=0xc000511760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005117e8 sp=0xc0005117e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 86 [GC worker (idle)]:
runtime.gopark(0x220a44da2168f, 0xc00018e100, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000511f60 sp=0xc000511f40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc000511fe0 sp=0xc000511f60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000511fe8 sp=0xc000511fe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 30 [GC worker (idle)]:
runtime.gopark(0x55e960751368, 0xc000682060, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050e760 sp=0xc00050e740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050e7e0 sp=0xc00050e760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050e7e8 sp=0xc00050e7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 87 [GC worker (idle)]:
runtime.gopark(0x55e960751368, 0xc000682080, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000512760 sp=0xc000512740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc0005127e0 sp=0xc000512760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005127e8 sp=0xc0005127e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 101 [GC worker (idle)]:
runtime.gopark(0x220a44da1fe81, 0xc0006820a0, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060df60 sp=0xc00060df40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060dfe0 sp=0xc00060df60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060dfe8 sp=0xc00060dfe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 102 [GC worker (idle)]:
runtime.gopark(0x55e960751368, 0xc0006820c0, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060e760 sp=0xc00060e740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060e7e0 sp=0xc00060e760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060e7e8 sp=0xc00060e7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 103 [GC worker (idle)]:
runtime.gopark(0x220a44da2176b, 0xc0006820e0, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060ef60 sp=0xc00060ef40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060efe0 sp=0xc00060ef60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060efe8 sp=0xc00060efe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 104 [GC worker (idle)]:
runtime.gopark(0x220a44da21776, 0xc000682100, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00060f760 sp=0xc00060f740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00060f7e0 sp=0xc00060f760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00060f7e8 sp=0xc00060f7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 31 [GC worker (idle)]:
runtime.gopark(0x55e960751368, 0xc00018e120, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050ef60 sp=0xc00050ef40 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050efe0 sp=0xc00050ef60 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050efe8 sp=0xc00050efe0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 32 [GC worker (idle)]:
runtime.gopark(0x55e960751368, 0xc00018e140, 0x18, 0x14, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc00050f760 sp=0xc00050f740 pc=0x55e95f0da016
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1200 +0xe5 fp=0xc00050f7e0 sp=0xc00050f760 pc=0x55e95f0bcce5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00050f7e8 sp=0xc00050f7e0 pc=0x55e95f10b3c1
created by runtime.gcBgMarkStartWorkers
/usr/lib/golang/src/runtime/mgc.go:1124 +0x25
goroutine 88 [select, locked to thread]:
runtime.gopark(0xc00050ffa8, 0x2, 0x90, 0xfe, 0xc00050ffa4)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000133e10 sp=0xc000133df0 pc=0x55e95f0da016
runtime.selectgo(0xc000133fa8, 0xc00050ffa0, 0x0, 0x0, 0x0, 0x1)
/usr/lib/golang/src/runtime/select.go:327 +0x772 fp=0xc000133f58 sp=0xc000133e10 pc=0x55e95f0e9e12
runtime.ensureSigM.func1()
/usr/lib/golang/src/runtime/signal_unix.go:890 +0x1a9 fp=0xc000133fe0 sp=0xc000133f58 pc=0x55e95f103ca9
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000133fe8 sp=0xc000133fe0 pc=0x55e95f10b3c1
created by runtime.ensureSigM
/usr/lib/golang/src/runtime/signal_unix.go:873 +0xbf
goroutine 114 [syscall]:
runtime.notetsleepg(0x0, 0x0)
/usr/lib/golang/src/runtime/lock_futex.go:236 +0x34 fp=0xc000e05fa0 sp=0xc000e05f68 pc=0x55e95f0ad6d4
os/signal.signal_recv()
/usr/lib/golang/src/runtime/sigqueue.go:169 +0x98 fp=0xc000e05fc0 sp=0xc000e05fa0 pc=0x55e95f1078d8
os/signal.loop()
/usr/lib/golang/src/os/signal/signal_unix.go:24 +0x19 fp=0xc000e05fe0 sp=0xc000e05fc0 pc=0x55e95f3ff919
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000e05fe8 sp=0xc000e05fe0 pc=0x55e95f10b3c1
created by os/signal.Notify.func1.1
/usr/lib/golang/src/os/signal/signal.go:151 +0x2c
goroutine 89 [select]:
runtime.gopark(0xc0006117b0, 0x2, 0x0, 0x0, 0xc0006116a4)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000e0fd08 sp=0xc000e0fce8 pc=0x55e95f0da016
runtime.selectgo(0xc000e0ffb0, 0xc0006116a0, 0x0, 0x0, 0x0, 0x1)
/usr/lib/golang/src/runtime/select.go:327 +0x772 fp=0xc000e0fe50 sp=0xc000e0fd08 pc=0x55e95f0e9e12
github.com/containers/podman/libpod/shutdown.Start.func1()
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:47 +0x99 fp=0xc000e0ffe0 sp=0xc000e0fe50 pc=0x55e95fc80c59
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000e0ffe8 sp=0xc000e0ffe0 pc=0x55e95f10b3c1
created by github.com/containers/podman/libpod/shutdown.Start
/builddir/build/BUILD/containers-podman-7f5e2fd/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:46 +0xea
Aborted (core dumped)
[root@nbapp817 ~]#
Created attachment 1928471 [details]
core file
Hi @bbaude Please let me know if you need anything else. We had another issue of podman panictoday after a system power cycle. We deleted /var/lib/containers/storage/libpod/bolt_state.db Removed all the stale conainers reported by 'podman ps --all --external' with Status as 'Storage' But podman image list still complained, about 'layer not known' # podman image ls Error: error retrieving label for image "b3a567473e54f9cdc3c0d8e45904e421c7c149e2cba5fe69a3327da17614b765": you may need to remove the image to resolve the error: layer not known After we removed all the images that were in error, 'podman image ls' reported empty # podman image ls REPOSITORY TAG IMAGE ID CREATED SIZE We see all layers present in /var/lib/containers/storage/overlay-layers/ We need a solution to fix this problem. Is there a way to reconstruct the image list. It sounds like we're starting to diverge here: We have the Podman database corruption issue we were originally discussing, but now we have another case of DB corruption, this time in c/storage. Can we keep the Podman database corruption issue (bolt_state.db) here, and open a fresh issue for the similar but separate c/storage corruption (layers.json). Can you also attach a corrupted layers.json (and any other files in the `/var/lib/containers/storage/overlay-layers/` directory that end in `.json`)? Finally: are we certain there are no container creation operations going on during steady-state? The layers file should only be modified when containers are created/deleted or images are pulled. (In reply to Matthew Heon from comment #22) > It sounds like we're starting to diverge here: We have the Podman database > corruption issue we were originally discussing, but now we have another case > of DB corruption, this time in c/storage. Can we keep the Podman database > corruption issue (bolt_state.db) here, and open a fresh issue for the > similar but separate c/storage corruption (layers.json). Can you also attach > a corrupted layers.json (and any other files in the > `/var/lib/containers/storage/overlay-layers/` directory that end in `.json`)? > > Finally: are we certain there are no container creation operations going on > during steady-state? The layers file should only be modified when containers > are created/deleted or images are pulled. Hello Matthew, We have a new case opened for the DB corruption in c/storage and we have the layers.json file available on the case as well. Since the current issue is different from the new corruption do you want me to open up a new Bug ? Regards, Carroline For opening a new bug: Yes, please do Mangesh, We have recently added support for sqlite into Podman in large part because of this bug. Unfortunately, we are unable to replicate this bug like you can (with boltdb); would you be willing to try a Podman where sqlite is used and see if you have better results? We are also adding a new feature which should also help this condition but it is not complete yet. |