Bug 2079759

Summary:	skopeo segfaults after rebuild with golang-1.18
Product:	Red Hat Enterprise Linux 8	Reporter:	Jindrich Novy <jnovy>
Component:	skopeo	Assignee:	Jindrich Novy <jnovy>
Status:	CLOSED ERRATA	QA Contact:	Yuhui Jiang <yujiang>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	8.7	CC:	bbaude, pthomas, tsweeney, umohnani, ypu
Target Milestone:	rc	Keywords:	Triaged
Target Release:	---	Flags:	pm-rhel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	skopeo-1.2.4-2.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2079763 (view as bug list)		Environment:
Last Closed:	2022-11-08 09:15:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2079763

Description Jindrich Novy 2022-04-28 08:02:49 UTC

This is happening in skopeo gating tests. The segfault might be in multiple components - skopeo/podman/runc so setting this to skopeo for now:

See: http://artifacts.osci.redhat.com/baseos-ci/redhat-module/15/03/15032/https___baseos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com-ci-artemis-mbs-sti/981/work-tests.yml0qzmAM/tests-E4mIyg/test.skopeo-root.bats.log

Log snippet:

not ok 9 copy: registry -> storage -> oci-archive
# (from function `die' in file ./helpers.bash, line 136,
#  from function `run_skopeo' in file ./helpers.bash, line 107,
#  in test file ./020-copy.bats, line 84)
#   `run_skopeo copy docker://$alpine containers-storage:$alpine' failed with status 134
# $ podman --runtime runc --cgroup-manager=cgroupfs run -d --name reg -v /tmp/skopeo_bats.tiP96a/auth:/auth:Z -p 5000:5000 quay.io/libpod/registry:2
# 0918375e0552b42ac8a915bc7a114f925e5a3c7449916f1bd2b5870243676f1d
# /usr/share/skopeo/test/system/./helpers.bash: connect: Connection refused
# /usr/share/skopeo/test/system/./helpers.bash: line 355: /dev/tcp/127.0.0.1/5000: Connection refused
# $ /usr/bin/skopeo copy docker://quay.io/libpod/alpine:latest containers-storage:quay.io/libpod/alpine:latest
# time="2022-04-27T21:23:57Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
# Getting image source signatures
# Copying blob sha256:9d16cba9fb961d1aafec9542f2bf7cb64acfc55245f9e4eb5abecd4cdc38d749
# Copying config sha256:961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
# Writing manifest to image destination
# Storing signatures
# unexpected fault address 0x0
# fatal error: fault
# [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x5623b5ea451f]
# 
# goroutine 1 [running]:
# runtime.throw({0x5623b6705788?, 0xc00003d920?})
# 	/usr/lib/golang/src/runtime/panic.go:992 +0x71 fp=0xc0000bbab8 sp=0xc0000bba88 pc=0x5623b5e77691
# runtime.sigpanic()
# 	/usr/lib/golang/src/runtime/signal_unix.go:825 +0x305 fp=0xc0000bbb08 sp=0xc0000bbab8 pc=0x5623b5e8ccc5
# aeshashbody()
# 	/usr/lib/golang/src/runtime/asm_amd64.s:1343 +0x39f fp=0xc0000bbb10 sp=0xc0000bbb08 pc=0x5623b5ea451f
# runtime.mapiternext(0xc00012a500)
# 	/usr/lib/golang/src/runtime/map.go:934 +0x2cb fp=0xc0000bbb80 sp=0xc0000bbb10 pc=0x5623b5e5246b
# runtime.mapiterinit(0xc0000bbc18?, 0x5623b65d95f6?, 0xc00044e2c0?)
# 	/usr/lib/golang/src/runtime/map.go:861 +0x228 fp=0xc0000bbba0 sp=0xc0000bbb80 pc=0x5623b5e52148
# reflect.mapiterinit(0xc0000bbc18?, 0x5623b65db0a7?, 0xc00012a500?)
# 	/usr/lib/golang/src/runtime/map.go:1373 +0x19 fp=0xc0000bbbc8 sp=0xc0000bbba0 pc=0x5623b5ea12b9
# github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/modern-go/reflect2/unsafe_map.go:112
# github.com/json-iterator/go.(*sortKeysMapEncoder).IsEmpty(0xc0000c1e10?, 0xc00027e2c0?)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/json-iterator/go/reflect_map.go:333 +0x28 fp=0xc0000bbc08 sp=0xc0000bbbc8 pc=0x5623b65cca68
# github.com/json-iterator/go.(*structFieldEncoder).IsEmpty(0xc00066e180, 0x5623b6ac080d?)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/json-iterator/go/reflect_struct_encoder.go:118 +0x42 fp=0xc0000bbc28 sp=0xc0000bbc08 pc=0x5623b65d97a2
# github.com/json-iterator/go.(*structEncoder).Encode(0xc00066e270, 0x82007fe982bb5108?, 0xc00054c7e0)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/json-iterator/go/reflect_struct_encoder.go:148 +0x565 fp=0xc0000bbd10 sp=0xc0000bbc28 pc=0x5623b65d9e05
# github.com/json-iterator/go.(*OptionalEncoder).Encode(0x5623b6b76be0?, 0xc0000bbde8?, 0xc0000bbe08?)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/json-iterator/go/reflect_optional.go:70 +0xa4 fp=0xc0000bbd60 sp=0xc0000bbd10 pc=0x5623b65d11c4
# github.com/json-iterator/go.(*sliceEncoder).Encode(0xc000120450, 0xc0004f6bb0, 0xc00054c7e0)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/json-iterator/go/reflect_slice.go:42 +0x302 fp=0xc0000bbe18 sp=0xc0000bbd60 pc=0x5623b65d1c62
# github.com/json-iterator/go.(*OptionalEncoder).Encode(0xc00021a780?, 0x0?, 0x0?)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/json-iterator/go/reflect_optional.go:70 +0xa4 fp=0xc0000bbe68 sp=0xc0000bbe18 pc=0x5623b65d11c4
# github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc00031cc70, 0xc0004f6bb0, 0xc0005116e0?)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/json-iterator/go/reflect.go:219 +0x82 fp=0xc0000bbea0 sp=0xc0000bbe68 pc=0x5623b65c4162
# github.com/json-iterator/go.(*Stream).WriteVal(0xc00054c7e0, {0x5623b6ad3380, 0xc0004f6bb0})
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/json-iterator/go/reflect.go:98 +0x158 fp=0xc0000bbf10 sp=0xc0000bbea0 pc=0x5623b65c3478
# github.com/json-iterator/go.(*frozenConfig).Marshal(0xc00021a780, {0x5623b6ad3380, 0xc0004f6bb0})
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/json-iterator/go/config.go:299 +0xc9 fp=0xc0000bbfa8 sp=0xc0000bbf10 pc=0x5623b65ba589
# github.com/containers/storage.(*layerStore).saveLayers(0xc0004f6b60)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/containers/storage/layers.go:482 +0x135 fp=0xc0000bc088 sp=0xc0000bbfa8 pc=0x5623b666fc35
# github.com/containers/storage.(*layerStore).Save(0xc0004f6b60)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/containers/storage/layers.go:465 +0xc5 fp=0xc0000bc0f8 sp=0xc0000bc088 pc=0x5623b666f985
# github.com/containers/storage.(*layerStore).Put(0xc0004f6b60, {0xc000502377, 0x40}, 0x0, {0x0, 0x0, 0xc0000bc590?}, {0x0, 0x0}, 0x0, ...)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/containers/storage/layers.go:797 +0x14a9 fp=0xc0000bc4e8 sp=0xc0000bc0f8 pc=0x5623b6673029
# github.com/containers/storage.(*store).PutLayer(0xc0000a2580, {0xc000502377, 0x40}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, 0x0}, ...)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/containers/storage/store.go:1047 +0xdef fp=0xc0000bc8e8 sp=0xc0000bc4e8 pc=0x5623b66810ef
# github.com/containers/image/v5/storage.(*storageImageDestination).commitLayer(0xc0000a1e60, {0x5623b6c68cf0, 0xc00012c000}, {{{0xc000044f50, 0x47}, 0x2c327e, {0x0, 0x0, 0x0}, 0x0, ...}, ...}, ...)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/containers/image/v5/storage/storage_image.go:922 +0xa68 fp=0xc0000bcb58 sp=0xc0000bc8e8 pc=0x5623b66a7088
# github.com/containers/image/v5/storage.(*storageImageDestination).Commit(0xc0000a1e60, {0x5623b6c68cf0, 0xc00012c000}, {0x5623b6c66c60?, 0xc0003b1980?})
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/containers/image/v5/storage/storage_image.go:970 +0x29b3 fp=0xc0000bd478 sp=0xc0000bcb58 pc=0x5623b66aa373
# github.com/containers/image/v5/copy.Image({0x5623b6c68cf0, 0xc00012c000}, 0x1c0?, {0x5623b6c6cc80, 0xc0001295c0}, {0x5623b6c6d280?, 0xc0001f89e0?}, 0x1c0?)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/containers/image/v5/copy/copy.go:355 +0x113d fp=0xc0000bd838 sp=0xc0000bd478 pc=0x5623b646781d
# main.(*copyOptions).run.func1()
# 	/builddir/build/BUILD/containers-skopeo-9921983/cmd/skopeo/copy.go:187 +0x17d fp=0xc0000bd968 sp=0xc0000bd838 pc=0x5623b66e8add
# github.com/containers/common/pkg/retry.RetryIfNecessary({0x5623b6c68cf0, 0xc00012c000}, 0xc0000bdc68, 0xc0001b4be0)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/containers/common/pkg/retry/retry.go:27 +0x4e fp=0xc0000bda48 sp=0xc0000bd968 pc=0x5623b616c1ae
# main.(*copyOptions).run(0xc0004d8240, {0xc000140d20?, 0x5623b66d3ea5?, 0x5623b66d72a5?}, {0x5623b6c62618, 0xc000126008})
# 	/builddir/build/BUILD/containers-skopeo-9921983/cmd/skopeo/copy.go:186 +0xb51 fp=0xc0000bdd18 sp=0xc0000bda48 pc=0x5623b66e8611
# main.(*copyOptions).run-fm({0xc000140d20?, 0x5623b6c62618?, 0xc000126008?}, {0x5623b6c62618?, 0xc000126008?})
# 	<autogenerated>:1 +0x47 fp=0xc0000bdd58 sp=0xc0000bdd18 pc=0x5623b66f8287
# main.commandAction.func1(0xc000341900?, {0xc000140d20, 0x2, 0x2})
# 	/builddir/build/BUILD/containers-skopeo-9921983/cmd/skopeo/utils.go:30 +0x71 fp=0xc0000bddb0 sp=0xc0000bdd58 pc=0x5623b66f6a91
# github.com/spf13/cobra.(*Command).execute(0xc000341900, {0xc000140cc0, 0x2, 0x2})
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/spf13/cobra/command.go:852 +0x67c fp=0xc0000bde88 sp=0xc0000bddb0 pc=0x5623b66d69bc
# github.com/spf13/cobra.(*Command).ExecuteC(0xc000341680)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/spf13/cobra/command.go:960 +0x39c fp=0xc0000bdf40 sp=0xc0000bde88 pc=0x5623b66d6f9c
# github.com/spf13/cobra.(*Command).Execute(...)
# 	/builddir/build/BUILD/containers-skopeo-9921983/vendor/src/github.com/spf13/cobra/command.go:897
# main.main()
# 	/builddir/build/BUILD/containers-skopeo-9921983/cmd/skopeo/main.go:103 +0x56 fp=0xc0000bdf80 sp=0xc0000bdf40 pc=0x5623b66ef776
# runtime.main()
# 	/usr/lib/golang/src/runtime/proc.go:250 +0x213 fp=0xc0000bdfe0 sp=0xc0000bdf80 pc=0x5623b5e79e73
# runtime.goexit()
# 	/usr/lib/golang/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000bdfe8 sp=0xc0000bdfe0 pc=0x5623b5ea71a1
# 
# goroutine 2 [force gc (idle)]:
# runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
# 	/usr/lib/golang/src/runtime/proc.go:361 +0xd6 fp=0xc000062fb0 sp=0xc000062f90 pc=0x5623b5e7a236
# runtime.goparkunlock(...)
# 	/usr/lib/golang/src/runtime/proc.go:367
# runtime.forcegchelper()
# 	/usr/lib/golang/src/runtime/proc.go:301 +0xad fp=0xc000062fe0 sp=0xc000062fb0 pc=0x5623b5e7a0cd
# runtime.goexit()
# 	/usr/lib/golang/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000062fe8 sp=0xc000062fe0 pc=0x5623b5ea71a1
# created by runtime.init.7
# 	/usr/lib/golang/src/runtime/proc.go:289 +0x25
# 
# goroutine 3 [GC sweep wait]:
# runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
# 	/usr/lib/golang/src/runtime/proc.go:361 +0xd6 fp=0xc000063790 sp=0xc000063770 pc=0x5623b5e7a236
# runtime.goparkunlock(...)
# 	/usr/lib/golang/src/runtime/proc.go:367
# runtime.bgsweep(0x0?)
# 	/usr/lib/golang/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc0000637c8 sp=0xc000063790 pc=0x5623b5e66437
# runtime.gcenable.func1()
# 	/usr/lib/golang/src/runtime/mgc.go:177 +0x26 fp=0xc0000637e0 sp=0xc0000637c8 pc=0x5623b5e5bfa6
# runtime.goexit()
# 	/usr/lib/golang/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000637e8 sp=0xc0000637e0 pc=0x5623b5ea71a1
# created by runtime.gcenable
# 	/usr/lib/golang/src/runtime/mgc.go:177 +0x6b
# 
# goroutine 4 [GC scavenge wait]:
# runtime.gopark(0x3ed1f000b8c?, 0x10000?, 0x0?, 0x0?, 0x0?)
# 	/usr/lib/golang/src/runtime/proc.go:361 +0xd6 fp=0xc000063f20 sp=0xc000063f00 pc=0x5623b5e7a236
# runtime.goparkunlock(...)
# 	/usr/lib/golang/src/runtime/proc.go:367
# runtime.bgscavenge(0x0?)
# 	/usr/lib/golang/src/runtime/mgcscavenge.go:364 +0x2a5 fp=0xc000063fc8 sp=0xc000063f20 pc=0x5623b5e64245
# runtime.gcenable.func2()
# 	/usr/lib/golang/src/runtime/mgc.go:178 +0x26 fp=0xc000063fe0 sp=0xc000063fc8 pc=0x5623b5e5bf46
# runtime.goexit()
# 	/usr/lib/golang/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x5623b5ea71a1
# created by runtime.gcenable
# 	/usr/lib/golang/src/runtime/mgc.go:178 +0xaa
# 
# goroutine 18 [finalizer wait]:
# runtime.gopark(0x0?, 0x5623b6c5cc08?, 0xe0?, 0x20?, 0x2000000020?)
# 	/usr/lib/golang/src/runtime/proc.go:361 +0xd6 fp=0xc000073e30 sp=0xc000073e10 pc=0x5623b5e7a236
# runtime.goparkunlock(...)
# 	/usr/lib/golang/src/runtime/proc.go:367
# runtime.runfinq()
# 	/usr/lib/golang/src/runtime/mfinal.go:177 +0xb3 fp=0xc000073fe0 sp=0xc000073e30 pc=0x5623b5e5afd3
# runtime.goexit()
# 	/usr/lib/golang/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x5623b5ea71a1
# created by runtime.createfing
# 	/usr/lib/golang/src/runtime/mfinal.go:157 +0x45
# 
# goroutine 19 [GC worker (idle)]:
# runtime.gopark(0x3ed3b22e251?, 0x0?, 0x0?, 0x0?, 0x0?)
# 	/usr/lib/golang/src/runtime/proc.go:361 +0xd6 fp=0xc00005e758 sp=0xc00005e738 pc=0x5623b5e7a236
# runtime.gcBgMarkWorker()
# 	/usr/lib/golang/src/runtime/mgc.go:1207 +0xe5 fp=0xc00005e7e0 sp=0xc00005e758 pc=0x5623b5e5e0c5
# runtime.goexit()
# 	/usr/lib/golang/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005e7e8 sp=0xc00005e7e0 pc=0x5623b5ea71a1
# created by runtime.gcBgMarkStartWorkers
# 	/usr/lib/golang/src/runtime/mgc.go:1131 +0x25
# 
# goroutine 5 [GC worker (idle)]:
# runtime.gopark(0x3ed3b22cdd1?, 0x0?, 0x0?, 0x0?, 0x0?)
# 	/usr/lib/golang/src/runtime/proc.go:361 +0xd6 fp=0xc000064758 sp=0xc000064738 pc=0x5623b5e7a236
# runtime.gcBgMarkWorker()
# 	/usr/lib/golang/src/runtime/mgc.go:1207 +0xe5 fp=0xc0000647e0 sp=0xc000064758 pc=0x5623b5e5e0c5
# runtime.goexit()
# 	/usr/lib/golang/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000647e8 sp=0xc0000647e0 pc=0x5623b5ea71a1
# created by runtime.gcBgMarkStartWorkers
# 	/usr/lib/golang/src/runtime/mgc.go:1131 +0x25
# timeout: the monitored command dumped core
# [ rc=134 (** EXPECTED 0 **) ]
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #| FAIL: exit code is 134; expected 0
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# 0918375e0552b42ac8a915bc7a114f925e5a3c7449916f1bd2b5870243676f1d

Comment 1 Tom Sweeney 2022-04-28 19:40:58 UTC

I'm betting this is the same fun Brent Baude has chased in Podman.  @bbaude thoughts?

Comment 2 Brent Baude 2022-04-29 02:48:13 UTC

yes, we confirmed that for Jindrich (as did Nalin)

Comment 3 Brent Baude 2022-04-29 02:48:43 UTC

if required, I could also handle it ... let me know tomorrow 4/29

Comment 14 errata-xmlrpc 2022-11-08 09:15:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7457