Description of problem: The current version and 2.56.2 breaks certbot so that it can no longer run or issue certificates. Version-Release number of selected component (if applicable): 2.56.2+ How reproducible: 100% of the time Steps to Reproduce: 1. upgrade snap 2. run certbot, it will crash Actual results: list cert: → certbot certificates fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0] runtime stack: runtime.throw({0x7257cd, 0x0}) /usr/lib/golang/src/runtime/panic.go:1198 +0x71 fp=0x7ffe1d6c4f20 sp=0x7ffe1d6c4ef0 pc=0x435291 runtime.sigpanic() /usr/lib/golang/src/runtime/signal_unix.go:719 +0x396 fp=0x7ffe1d6c4f70 sp=0x7ffe1d6c4f20 pc=0x44aa16 goroutine 1 [syscall, locked to thread]: runtime.cgocall(0x5e9a70, 0xc0000d96c0) /usr/lib/golang/src/runtime/cgocall.go:156 +0x5c fp=0xc0000d9698 sp=0xc0000d9660 pc=0x40551c crypto/internal/boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL() _cgo_gotypes.go:603 +0x49 fp=0xc0000d96c0 sp=0xc0000d9698 pc=0x5366c9 crypto/internal/boring.init.0() /usr/lib/golang/src/crypto/internal/boring/boring.go:46 +0x45 fp=0xc0000d96f8 sp=0xc0000d96c0 pc=0x537445 runtime.doInit(0xad84a0) /usr/lib/golang/src/runtime/proc.go:6498 +0x123 fp=0xc0000d9830 sp=0xc0000d96f8 pc=0x4449a3 runtime.doInit(0xad61a0) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9968 sp=0xc0000d9830 pc=0x4448f1 runtime.doInit(0xad9280) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9aa0 sp=0xc0000d9968 pc=0x4448f1 runtime.doInit(0xad8260) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9bd8 sp=0xc0000d9aa0 pc=0x4448f1 runtime.doInit(0xad9bc0) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9d10 sp=0xc0000d9bd8 pc=0x4448f1 runtime.doInit(0xad7220) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9e48 sp=0xc0000d9d10 pc=0x4448f1 runtime.doInit(0xad7fe0) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9f80 sp=0xc0000d9e48 pc=0x4448f1 runtime.main() /usr/lib/golang/src/runtime/proc.go:238 +0x1e6 fp=0xc0000d9fe0 sp=0xc0000d9f80 pc=0x437926 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000d9fe8 sp=0xc0000d9fe0 pc=0x464361 goroutine 2 [force gc (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000036fb0 sp=0xc000036f90 pc=0x437d36 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.forcegchelper() /usr/lib/golang/src/runtime/proc.go:306 +0xad fp=0xc000036fe0 sp=0xc000036fb0 pc=0x437bcd runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000036fe8 sp=0xc000036fe0 pc=0x464361 created by runtime.init.8 /usr/lib/golang/src/runtime/proc.go:294 +0x25 goroutine 3 [GC sweep wait]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc0000377b0 sp=0xc000037790 pc=0x437d36 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.bgsweep() /usr/lib/golang/src/runtime/mgcsweep.go:163 +0x88 fp=0xc0000377e0 sp=0xc0000377b0 pc=0x4243a8 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000377e8 sp=0xc0000377e0 pc=0x464361 created by runtime.gcenable /usr/lib/golang/src/runtime/mgc.go:181 +0x55 goroutine 4 [GC scavenge wait]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000037f80 sp=0xc000037f60 pc=0x437d36 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.bgscavenge() /usr/lib/golang/src/runtime/mgcscavenge.go:265 +0xcd fp=0xc000037fe0 sp=0xc000037f80 pc=0x4224ad runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000037fe8 sp=0xc000037fe0 pc=0x464361 created by runtime.gcenable /usr/lib/golang/src/runtime/mgc.go:182 +0x65 goroutine 18 [finalizer wait]: runtime.gopark(0xaec640, 0xc000036770, 0xf1, 0x48, 0xad75e0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000036630 sp=0xc000036610 pc=0x437d36 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.runfinq() /usr/lib/golang/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000367e0 sp=0xc000036630 pc=0x419f33 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000367e8 sp=0xc0000367e0 pc=0x464361 created by runtime.createfing /usr/lib/golang/src/runtime/mfinal.go:157 +0x45 Aborted Installing certbot → snap install certbot --classic error: cannot perform the following tasks: - Run hook prepare-plug-plugin of snap "certbot" (run hook "prepare-plug-plugin": ----- fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0] runtime stack: runtime.throw({0x7257cd, 0x0}) /usr/lib/golang/src/runtime/panic.go:1198 +0x71 fp=0x7ffeb7291e90 sp=0x7ffeb7291e60 pc=0x435291 runtime.sigpanic() /usr/lib/golang/src/runtime/signal_unix.go:719 +0x396 fp=0x7ffeb7291ee0 sp=0x7ffeb7291e90 pc=0x44aa16 goroutine 1 [syscall, locked to thread]: runtime.cgocall(0x5e9a70, 0xc0000db6c0) /usr/lib/golang/src/runtime/cgocall.go:156 +0x5c fp=0xc0000db698 sp=0xc0000db660 pc=0x40551c crypto/internal/boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL() _cgo_gotypes.go:603 +0x49 fp=0xc0000db6c0 sp=0xc0000db698 pc=0x5366c9 crypto/internal/boring.init.0() /usr/lib/golang/src/crypto/internal/boring/boring.go:46 +0x45 fp=0xc0000db6f8 sp=0xc0000db6c0 pc=0x537445 runtime.doInit(0xad84a0) /usr/lib/golang/src/runtime/proc.go:6498 +0x123 fp=0xc0000db830 sp=0xc0000db6f8 pc=0x4449a3 runtime.doInit(0xad61a0) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000db968 sp=0xc0000db830 pc=0x4448f1 runtime.doInit(0xad9280) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000dbaa0 sp=0xc0000db968 pc=0x4448f1 runtime.doInit(0xad8260) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000dbbd8 sp=0xc0000dbaa0 pc=0x4448f1 runtime.doInit(0xad9bc0) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000dbd10 sp=0xc0000dbbd8 pc=0x4448f1 runtime.doInit(0xad7220) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000dbe48 sp=0xc0000dbd10 pc=0x4448f1 runtime.doInit(0xad7fe0) /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000dbf80 sp=0xc0000dbe48 pc=0x4448f1 runtime.main() /usr/lib/golang/src/runtime/proc.go:238 +0x1e6 fp=0xc0000dbfe0 sp=0xc0000dbf80 pc=0x437926 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000dbfe8 sp=0xc0000dbfe0 pc=0x464361 goroutine 2 [force gc (idle)]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000032fb0 sp=0xc000032f90 pc=0x437d36 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.forcegchelper() /usr/lib/golang/src/runtime/proc.go:306 +0xad fp=0xc000032fe0 sp=0xc000032fb0 pc=0x437bcd runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000032fe8 sp=0xc000032fe0 pc=0x464361 created by runtime.init.8 /usr/lib/golang/src/runtime/proc.go:294 +0x25 goroutine 3 [GC sweep wait]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc0000337b0 sp=0xc000033790 pc=0x437d36 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.bgsweep() /usr/lib/golang/src/runtime/mgcsweep.go:163 +0x88 fp=0xc0000337e0 sp=0xc0000337b0 pc=0x4243a8 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000337e8 sp=0xc0000337e0 pc=0x464361 created by runtime.gcenable /usr/lib/golang/src/runtime/mgc.go:181 +0x55 goroutine 4 [GC scavenge wait]: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000033f80 sp=0xc000033f60 pc=0x437d36 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.bgscavenge() /usr/lib/golang/src/runtime/mgcscavenge.go:265 +0xcd fp=0xc000033fe0 sp=0xc000033f80 pc=0x4224ad runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000033fe8 sp=0xc000033fe0 pc=0x464361 created by runtime.gcenable /usr/lib/golang/src/runtime/mgc.go:182 +0x65 goroutine 18 [finalizer wait]: runtime.gopark(0xaec640, 0xc000032770, 0xf1, 0x48, 0xad75e0) /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000032630 sp=0xc000032610 pc=0x437d36 runtime.goparkunlock(...) /usr/lib/golang/src/runtime/proc.go:372 runtime.runfinq() /usr/lib/golang/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000327e0 sp=0xc000032630 pc=0x419f33 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000327e8 sp=0xc0000327e0 pc=0x464361 created by runtime.createfing /usr/lib/golang/src/runtime/mfinal.go:157 +0x45 -----) Expected results: Additional info: I reported the issue here https://github.com/certbot/certbot/issues/9332 for snapd 2.56.2 https://forum.snapcraft.io/t/el7-snap-2-56-2-causes-certbot-to-fail/30669
AFAICT this is caused by boringcrypto, a FIPS compliant build of Go that is used in EPEL. Those binaries are built statically as they are executed inside the snap's mount namespace. Apparently boringcrypto mode still tries to dlopen() openssl, and probably trips over the openssl binary from the base of the snap. I can try to disable boringcrypto for those specific binaries. Ideally, boringcrypto should not break Go.
Why is this only an issue on EL7 though? I believe I saw mention of boringcrypto in the previous posts I made about this.
*** Bug 2153040 has been marked as a duplicate of this bug. ***
Digging a bit, dlopen(libcrypto.so) with default Go linker flags works fine, but it crashes if you build the Go program with -static (like is done for snap-exec, snapctl, snap-update-ns). The crash itself is produced inside an `fopen` call in RHEL's `openssl-1.0.2j-deprecate-algos.patch`, but I think even in the absence of that, something else would trigger a crash soon after anyway. I noticed an "no_openssl" Go build tag in the Go sources, which isn't part of mainline Go but it's in the (now deprecated) FIPS fork. I tried this build tag for the static binaries and placing them on top of the current rpm, I am able to install and run Certbot (and other classic snaps) successfully 🎉. @Maciek Borzecki what are your thoughts about rebuilding the affected packages with these tags and making a release? We have a large number of users affected by this issue at the moment. Thank you! --- One worrying note, I think the programs are still unsafe due to the way that glibc is linked statically. There are warnings in the build about this: # github.com/snapcore/snapd/cmd/snapctl /tmp/go-link-605309214/000008.o: In function `mygetgrouplist': /_/os/user/getgrouplist_unix.go:15: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/go-link-605309214/000007.o: In function `mygetgrgid_r': /_/os/user/cgo_lookup_unix.go:37: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/go-link-605309214/000007.o: In function `mygetgrnam_r': /_/os/user/cgo_lookup_unix.go:42: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/go-link-605309214/000007.o: In function `mygetpwnam_r': /_/os/user/cgo_lookup_unix.go:32: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/go-link-605309214/000007.o: In function `mygetpwuid_r': /_/os/user/cgo_lookup_unix.go:27: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/go-link-605309214/000004.o: In function `_cgo_6cc2654a8ed3_C2func_getaddrinfo': /tmp/go-build/cgo-gcc-prolog:58: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking I suspect that any code path which hits these functions might crash the binaries again. But, at least classic snap support seems to work again.
(In reply to Alex Zorin from comment #4) > Digging a bit, dlopen(libcrypto.so) with default Go linker flags works fine, > but it crashes if you build the Go program with -static (like is done for > snap-exec, snapctl, snap-update-ns). > > The crash itself is produced inside an `fopen` call in RHEL's > `openssl-1.0.2j-deprecate-algos.patch`, but I think even in the absence of > that, something else would trigger a crash soon after anyway. > > I noticed an "no_openssl" Go build tag in the Go sources, which isn't part > of mainline Go but it's in the (now deprecated) FIPS fork. > > I tried this build tag for the static binaries and placing them on top of > the current rpm, I am able to install and run Certbot (and other classic > snaps) successfully 🎉. > > @Maciek Borzecki what are your thoughts about rebuilding the affected > packages with these tags and making a release? We have a large number of > users affected by this issue at the moment. Thank you! Thank you for looking into this! I'll try to push out some builds today. > > --- > > One worrying note, I think the programs are still unsafe due to the way that > glibc is linked statically. There are warnings in the build about this: > > # github.com/snapcore/snapd/cmd/snapctl > /tmp/go-link-605309214/000008.o: In function `mygetgrouplist': > /_/os/user/getgrouplist_unix.go:15: warning: Using 'getgrouplist' in > statically linked applications requires at runtime the shared libraries from > the glibc version used for linking > /tmp/go-link-605309214/000007.o: In function `mygetgrgid_r': > /_/os/user/cgo_lookup_unix.go:37: warning: Using 'getgrgid_r' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > /tmp/go-link-605309214/000007.o: In function `mygetgrnam_r': > /_/os/user/cgo_lookup_unix.go:42: warning: Using 'getgrnam_r' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > /tmp/go-link-605309214/000007.o: In function `mygetpwnam_r': > /_/os/user/cgo_lookup_unix.go:32: warning: Using 'getpwnam_r' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > /tmp/go-link-605309214/000007.o: In function `mygetpwuid_r': > /_/os/user/cgo_lookup_unix.go:27: warning: Using 'getpwuid_r' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > /tmp/go-link-605309214/000004.o: In function > `_cgo_6cc2654a8ed3_C2func_getaddrinfo': > /tmp/go-build/cgo-gcc-prolog:58: warning: Using 'getaddrinfo' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > > I suspect that any code path which hits these functions might crash the > binaries again. But, at least classic snap support seems to work again. This is fine. AFAIK the binaries in question do not execute any code that would trigger these calls.
FEDORA-2022-8f35a2f4f2 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-8f35a2f4f2
FEDORA-2022-8f35a2f4f2 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2022-4790972cca has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-4790972cca
FEDORA-EPEL-2022-80b8c04562 has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-80b8c04562
FEDORA-EPEL-2022-548bcdf1ae has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-548bcdf1ae
FEDORA-EPEL-2022-548bcdf1ae has been pushed to the Fedora EPEL 8 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-548bcdf1ae See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2022-4790972cca has been pushed to the Fedora EPEL 7 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-4790972cca See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2022-80b8c04562 has been pushed to the Fedora EPEL 9 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-80b8c04562 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2022-4790972cca has been pushed to the Fedora EPEL 7 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2022-548bcdf1ae has been pushed to the Fedora EPEL 8 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2022-80b8c04562 has been pushed to the Fedora EPEL 9 stable repository. If problem still persists, please make note of it in this bug report.