There's a problem that's cropped up now that -Wl,--as-needed is included in LDFLAGS by default. libpcreposix has symbols with the same names as those in libc (regcomp, regexec, regerror, regfree) and using --as-needed will change the symbol resolution order, causing fun breakage. It took me some time to figure out that this is the cause of some massive instability in a package I help maintain, cyrus-imapd. While trying to figure out why others don't see the same problem, I found that other distros which use --as-needed by default also patch pcre so that this problem doesn't exist. For example: The Debian patch, which I assume goes for Ubuntu as well: https://sources.debian.org/src/pcre3/2:8.39-11/debian/patches/pcreposix.patch/ OpenMandriva: https://github.com/OpenMandrivaAssociation/pcre/blob/master/pcre-pcreposix-glibc-conflict.patch Mageia: http://svnweb.mageia.org/packages/cauldron/pcre/current/SOURCES/pcre-pcreposix-glibc-conflict.patch?view=markup The Suse spec, which uses the debian patch: https://build.opensuse.org/package/view_file/openSUSE:Factory/pcre/pcre.spec Arch doesn't appear to patch pcre: https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/pcre So... any thoughts on whether we should be applying this patch in Fedora? Or maybe there some specific way to avoid the breakage with --as-needed that I should pass onto my upstream.
I already replied at <https://github.com/cyrusimap/cyrus-imapd/issues/2629#issuecomment-455993225>. Simply I cannot add that patch because it breaks ABI. I could do that in Rawhide only. However, I can add pcre_regcomp etc. to the pcreposix library and header file, and users can change their applications to use that functions explicitly. Though I'm not sure if that helps in preventing from mixing the symbols. I need to test it. Do you have a minimal reproducer? Or build logs from your affected cyrus-imapd? I'd like to see the exact linker flags cyrus-imapd uses. At the end you should recommend your upstream to use PCRE API directly instead of relying in the pcreposix emulation. Especially if the intention is to use PCRE's implementation instead of glibc's one. And actually upstream should use PCRE2. PCRE is not developed anymore. Or if upstream does not insist on PCRE's implementation, then it should stop linking to and rely on libc's instead.
The crash itself is caused by regex_t that has a different size in glibc (64 B) and in pcreposix (24 B). If a program includes <pcreposix.h>, but is not linked to pcreposix library at run-time, glibc's implementation touches memory out of the structure and the program can crash. A reproducer: #ifdef PCRE #include <pcreposix.h> #else #include <sys/types.h> #include <regex.h> #endif int main(void) { regex_t compiled; if (0 == regcomp(&compiled, "a", 0)) { regfree(&compiled); } return 0; } That means once a program includes <pcreposix.h>, it must link to the pcreposix library. PCRE2 (a PCRE successor) added a prefixed functions (e.g. pcre2_regcomp()) as a result of a similar bug report and documents that it's up to the application to ensure the proper linking (e.g. by redefining the symbols with a -Dregcomp=pcre2_regcomp preprocessor option). After studying this issue I believe it's better to do the redefinition on PCRE side: (1) Add prefixed functions to the pcreposix library and header file. (2) Redefine nonprefixed functions to prefixed ones in the header file. Then rebuilding affected application should be enough to fix the issue. This fix is backward compatible. The issue with -Wl,--as-needed you observe depends on lazy binding in dlopen() probably, because building an executable with -Wl,--as-needed does not exclude pcreposix library from linkage: $ gcc -Wl,--as-needed -O0 -g -DPCRE test.c $(pcre-config --libs-posix) $ scanelf -n ./a.out TYPE NEEDED FILE ET_EXEC libpcreposix.so.0,libc.so.6 ./a.out This can be some inconsistency in glibc's dynamic linker. ld-linux.so reports the pcreposix library as unused: $ ldd --unused ./a.out Unused direct dependencies: /lib64/libpcreposix.so.0 But it actually uses it when linking the ELF: $ LD_DEBUG=bindings ./a.out 2>&1 | grep 'binding file \./a\.out.* to .*pcreposix' 4090: binding file ./a.out [0] to /lib64/libpcreposix.so.0 [0]: normal symbol `regcomp' 4090: binding file ./a.out [0] to /lib64/libpcreposix.so.0 [0]: normal symbol `regfree' This issue should disappear after I apply this fix.
I assume that all consumers of libpcreposix will need to recompile immediately. Looks like there aren't all that many: Falcon Io-language Pound cyrus-imapd haproxy mariadb openCOLLADA privoxy scilab sslh vdr-epgfixer And I'm sorry I didn't see your request for a reproducer; yesterday was a US holiday so I wasn't reading email. Sadly I don't think I could have provided one anyway as I've just barely managed to understand what's actually broken. Thanks for figuring it out.
This change is implemented in pcre-8.42-7.fc30. This change will be ported to older Fedoras after some testing period. This change is backward compatible but not forward compatible. That means application built against the patched libpcreposix.so.0 library wont run against previous builds. This also apply to moving binary applications among distributions. However, since many of them already apply a similar patch, they had not been compatible even before. This change does not fix hijacking unprefixed symbols from glibc to pcre. However, that should not posses a significant risk because pcre's regex_t fits into glibc's one. If this assumption proved false, we can remove the unprefixed symbols later (together with bumping soname). I will rebuild the reverse dependencies. Please note that cyrus-imapd still needs some adjustments (bug #1668723).
I rebuilt all affected packages (actually not all from the list use regexp/pcreposix functions. Various projects simply overlink). If there are no regressions I will patch pcre and pcre2 in older Fedoras.
The patch was applied to pcre2-10.32-8.fc29 and pcre2-10.32-8.fc28.
pcre-8.43-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-96b27e298c
pcre-8.43-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-9965db700b
pcre-8.43-1.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-9965db700b
pcre-8.43-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-96b27e298c
pcre-8.43-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.
pcre-8.43-1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.