Bug 2152903
Summary: | 2.57.6 snapd on EL7 breaks certbot | ||
---|---|---|---|
Product: | [Fedora] Fedora EPEL | Reporter: | Danie de Jager <danie.dejager> |
Component: | snapd | Assignee: | Zygmunt Krynicki <me> |
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | urgent | Docs Contact: | |
Priority: | unspecified | ||
Version: | epel7 | CC: | bmw, colinsauze+redhatbugs, go-sig, jfronza, maciek.borzecki, me, ngompa13, rhbugs, whaidinger |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | snapd-2.57.6-2.fc38 snapd-2.57.6-2.el7 snapd-2.57.6-2.el8 snapd-2.57.6-2.el9 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2022-12-16 08:44:41 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Danie de Jager
2022-12-13 13:51:14 UTC
AFAICT this is caused by boringcrypto, a FIPS compliant build of Go that is used in EPEL. Those binaries are built statically as they are executed inside the snap's mount namespace. Apparently boringcrypto mode still tries to dlopen() openssl, and probably trips over the openssl binary from the base of the snap. I can try to disable boringcrypto for those specific binaries. Ideally, boringcrypto should not break Go. Why is this only an issue on EL7 though? I believe I saw mention of boringcrypto in the previous posts I made about this. *** Bug 2153040 has been marked as a duplicate of this bug. *** Digging a bit, dlopen(libcrypto.so) with default Go linker flags works fine, but it crashes if you build the Go program with -static (like is done for snap-exec, snapctl, snap-update-ns). The crash itself is produced inside an `fopen` call in RHEL's `openssl-1.0.2j-deprecate-algos.patch`, but I think even in the absence of that, something else would trigger a crash soon after anyway. I noticed an "no_openssl" Go build tag in the Go sources, which isn't part of mainline Go but it's in the (now deprecated) FIPS fork. I tried this build tag for the static binaries and placing them on top of the current rpm, I am able to install and run Certbot (and other classic snaps) successfully 🎉. @Maciek Borzecki what are your thoughts about rebuilding the affected packages with these tags and making a release? We have a large number of users affected by this issue at the moment. Thank you! --- One worrying note, I think the programs are still unsafe due to the way that glibc is linked statically. There are warnings in the build about this: # github.com/snapcore/snapd/cmd/snapctl /tmp/go-link-605309214/000008.o: In function `mygetgrouplist': /_/os/user/getgrouplist_unix.go:15: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/go-link-605309214/000007.o: In function `mygetgrgid_r': /_/os/user/cgo_lookup_unix.go:37: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/go-link-605309214/000007.o: In function `mygetgrnam_r': /_/os/user/cgo_lookup_unix.go:42: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/go-link-605309214/000007.o: In function `mygetpwnam_r': /_/os/user/cgo_lookup_unix.go:32: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/go-link-605309214/000007.o: In function `mygetpwuid_r': /_/os/user/cgo_lookup_unix.go:27: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/go-link-605309214/000004.o: In function `_cgo_6cc2654a8ed3_C2func_getaddrinfo': /tmp/go-build/cgo-gcc-prolog:58: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking I suspect that any code path which hits these functions might crash the binaries again. But, at least classic snap support seems to work again. (In reply to Alex Zorin from comment #4) > Digging a bit, dlopen(libcrypto.so) with default Go linker flags works fine, > but it crashes if you build the Go program with -static (like is done for > snap-exec, snapctl, snap-update-ns). > > The crash itself is produced inside an `fopen` call in RHEL's > `openssl-1.0.2j-deprecate-algos.patch`, but I think even in the absence of > that, something else would trigger a crash soon after anyway. > > I noticed an "no_openssl" Go build tag in the Go sources, which isn't part > of mainline Go but it's in the (now deprecated) FIPS fork. > > I tried this build tag for the static binaries and placing them on top of > the current rpm, I am able to install and run Certbot (and other classic > snaps) successfully 🎉. > > @Maciek Borzecki what are your thoughts about rebuilding the affected > packages with these tags and making a release? We have a large number of > users affected by this issue at the moment. Thank you! Thank you for looking into this! I'll try to push out some builds today. > > --- > > One worrying note, I think the programs are still unsafe due to the way that > glibc is linked statically. There are warnings in the build about this: > > # github.com/snapcore/snapd/cmd/snapctl > /tmp/go-link-605309214/000008.o: In function `mygetgrouplist': > /_/os/user/getgrouplist_unix.go:15: warning: Using 'getgrouplist' in > statically linked applications requires at runtime the shared libraries from > the glibc version used for linking > /tmp/go-link-605309214/000007.o: In function `mygetgrgid_r': > /_/os/user/cgo_lookup_unix.go:37: warning: Using 'getgrgid_r' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > /tmp/go-link-605309214/000007.o: In function `mygetgrnam_r': > /_/os/user/cgo_lookup_unix.go:42: warning: Using 'getgrnam_r' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > /tmp/go-link-605309214/000007.o: In function `mygetpwnam_r': > /_/os/user/cgo_lookup_unix.go:32: warning: Using 'getpwnam_r' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > /tmp/go-link-605309214/000007.o: In function `mygetpwuid_r': > /_/os/user/cgo_lookup_unix.go:27: warning: Using 'getpwuid_r' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > /tmp/go-link-605309214/000004.o: In function > `_cgo_6cc2654a8ed3_C2func_getaddrinfo': > /tmp/go-build/cgo-gcc-prolog:58: warning: Using 'getaddrinfo' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > > I suspect that any code path which hits these functions might crash the > binaries again. But, at least classic snap support seems to work again. This is fine. AFAIK the binaries in question do not execute any code that would trigger these calls. FEDORA-2022-8f35a2f4f2 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-8f35a2f4f2 FEDORA-2022-8f35a2f4f2 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report. FEDORA-EPEL-2022-4790972cca has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-4790972cca FEDORA-EPEL-2022-80b8c04562 has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-80b8c04562 FEDORA-EPEL-2022-548bcdf1ae has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-548bcdf1ae FEDORA-EPEL-2022-548bcdf1ae has been pushed to the Fedora EPEL 8 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-548bcdf1ae See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-EPEL-2022-4790972cca has been pushed to the Fedora EPEL 7 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-4790972cca See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-EPEL-2022-80b8c04562 has been pushed to the Fedora EPEL 9 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-80b8c04562 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-EPEL-2022-4790972cca has been pushed to the Fedora EPEL 7 stable repository. If problem still persists, please make note of it in this bug report. FEDORA-EPEL-2022-548bcdf1ae has been pushed to the Fedora EPEL 8 stable repository. If problem still persists, please make note of it in this bug report. FEDORA-EPEL-2022-80b8c04562 has been pushed to the Fedora EPEL 9 stable repository. If problem still persists, please make note of it in this bug report. |