Description of problem: prosody fail to start after a upgrade to F34 Version-Release number of selected component (if applicable): # rpm -q prosody lua prosody-0.11.8-2.fc34.x86_64 lua-5.4.2-2.fc34.x86_64 How reproducible: each time (on my instance) Steps to Reproduce: 1. try to start prosody 2. 3. Actual results: Apr 27 20:05:17 karona prosody[2514]: terminate called after throwing an instance of 'std::system_error' Apr 27 20:05:17 karona prosody[2514]: what(): Unknown error -1 Expected results: no error Additional info: I have a complete traceback. I will add comment as I find more information.
Apr 27 20:05:18 karona systemd-coredump[2516]: [🡕] Process 2514 (prosody) of user 993 dumped core. Stack trace of thread 2514: #0 0x00007f0e521e2292 raise (libc.so.6 + 0x3d292) #1 0x00007f0e521cb8a4 abort (libc.so.6 + 0x268a4) #2 0x00007f0e5019fa46 _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold (libstdc++.so.6 + 0xa1a46) #3 0x00007f0e501ab28c _ZN10__cxxabiv111__terminateEPFvvE (libstdc++.so.6 + 0xad28c) #4 0x00007f0e501ab2f7 _ZSt9terminatev (libstdc++.so.6 + 0xad2f7) #5 0x00007f0e501ab599 __cxa_throw (libstdc++.so.6 + 0xad599) #6 0x00007f0e501a2830 _ZSt20__throw_system_errori (libstdc++.so.6 + 0xa4830) #7 0x00007f0e51e9efff _ZN6icu_6720umtx_initImplPreInitERNS_9UInitOnceE (libicuuc.so.67 + 0x68fff) #8 0x00007f0e51f6c8ff usprep_open_67 (libicuuc.so.67 + 0x1368ff) #9 0x00007f0e52022f8a init_icu (encodings.so + 0x1f8a) #10 0x00007f0e52023051 luaopen_util_encodings (encodings.so + 0x2051) #11 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3) #12 0x00007f0e523e352b ccall.lto_priv.0 (liblua-5.4.so + 0x1752b) #13 0x00007f0e523d90a4 lua_callk (liblua-5.4.so + 0xd0a4) #14 0x00007f0e523f466b ll_require (liblua-5.4.so + 0x2866b) #15 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3) #16 0x00007f0e523dd82d f_call (liblua-5.4.so + 0x1182d) #17 0x00007f0e523e2853 luaD_rawrunprotected (liblua-5.4.so + 0x16853) #18 0x00007f0e523e5022 luaD_pcall (liblua-5.4.so + 0x19022) #19 0x00007f0e523d921c lua_pcallk (liblua-5.4.so + 0xd21c) #20 0x00007f0e523dec45 luaB_pcall (liblua-5.4.so + 0x12c45) #21 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3) #22 0x00007f0e523f97ed luaV_execute (liblua-5.4.so + 0x2d7ed) #23 0x00007f0e523e3544 ccall.lto_priv.0 (liblua-5.4.so + 0x17544) #24 0x00007f0e523d90a4 lua_callk (liblua-5.4.so + 0xd0a4) #25 0x00007f0e523f466b ll_require (liblua-5.4.so + 0x2866b) #26 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3) #27 0x00007f0e523f97ed luaV_execute (liblua-5.4.so + 0x2d7ed) #28 0x00007f0e523e3544 ccall.lto_priv.0 (liblua-5.4.so + 0x17544) #29 0x00007f0e523d90a4 lua_callk (liblua-5.4.so + 0xd0a4) #30 0x00007f0e523f466b ll_require (liblua-5.4.so + 0x2866b) #31 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3) #32 0x00007f0e523f97ed luaV_execute (liblua-5.4.so + 0x2d7ed) #33 0x00007f0e523dd846 f_call (liblua-5.4.so + 0x11846) #34 0x00007f0e523e2853 luaD_rawrunprotected (liblua-5.4.so + 0x16853) #35 0x00007f0e523e5022 luaD_pcall (liblua-5.4.so + 0x19022) #36 0x00007f0e523d921c lua_pcallk (liblua-5.4.so + 0xd21c) #37 0x000055716f6f4a4f docall (lua + 0x2a4f) #38 0x000055716f6f5607 pmain (lua + 0x3607) #39 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3) #40 0x00007f0e523dd82d f_call (liblua-5.4.so + 0x1182d) #41 0x00007f0e523e2853 luaD_rawrunprotected (liblua-5.4.so + 0x16853) #42 0x00007f0e523e5022 luaD_pcall (liblua-5.4.so + 0x19022) #43 0x00007f0e523d921c lua_pcallk (liblua-5.4.so + 0xd21c) #44 0x000055716f6f470f main (lua + 0x270f) #45 0x00007f0e521ccb75 __libc_start_main (libc.so.6 + 0x27b75) #46 0x000055716f6f47be _start (lua + 0x27be) So this is a issue with icu_67::umtx_initImplPreInit, it seems. And prosodyctl also fail: # prosodyctl terminate called after throwing an instance of 'std::system_error' what(): Unknown error -1 Aborted (core dumped)
So somewhere here: https://hg.prosody.im/trunk/file/tip/util-src/encodings.c#l342
So prosodyctl work on a fresh installation. This is quite curious.
So to be precise, the line that fail is https://hg.prosody.im/trunk/file/tip/util-src/encodings.c#l345 #11 usprep_getProfile (status=0x7fffffffd7a4, name=0x7ffff7b7c40c "rfc3491", path=0x0) at /usr/src/debug/icu-67.1-6.fc34.x86_64/source/common/usprep.cpp:311 #12 usprep_open_67 (path=0x0, name=0x7ffff7b7c40c "rfc3491", status=0x7fffffffd7a4) at /usr/src/debug/icu-67.1-6.fc34.x86_64/source/common/usprep.cpp:401
Given prosody and prosodyctl have both been working fine while the package was built [1][2], prosody might be the victim of later introduced breaking changes in Fedora (maybe Lua or ICU)...assuming that you are talking about an empty, unconfigured prosody here. [1] https://kojipkgs.fedoraproject.org//packages/prosody/0.11.8/2.fc34/data/logs/x86_64/build.log [2] https://src.fedoraproject.org/rpms/prosody/c/9354a84e62995723d190b95b85072541b26f3a6f?branch=f34
Ok so #3 was wrong, I forgot I didn't upgrade this specific server. On F34, the shortest reproducer I have is the following: # cat /tmp/e.lua package.path = '/usr/lib64/prosody/?.lua;'; package.cpath = '/usr/lib64/prosody/?.so;'; local encodings = require "util.encodings"; # lua /tmp/e.lua terminate called after throwing an instance of 'std::system_error' what(): Unknown error -1 Aborted (core dumped) Running gdb show something in libicu fail.
I am trying a scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=66813543
Fail as well: https://koji.fedoraproject.org/koji/getfile?taskID=66813654&volume=DEFAULT&name=build.log&offset=-4000 ++ pwd + export PROSODY_DATADIR=/builddir/build/BUILD/prosody-0.11.8/tests/data + PROSODY_DATADIR=/builddir/build/BUILD/prosody-0.11.8/tests/data + ./prosodyctl about terminate called after throwing an instance of 'std::system_error' what(): Unknown error -1 /var/tmp/rpm-tmp.V9fT0d: line 47: 55130 Aborted (core dumped) ./prosodyctl about
Switching from icu to idn fixed the problem: https://koji.fedoraproject.org/koji/taskinfo?taskID=66824480 That's a workaround, and to be honest, I do not know the impact. Using ICU was added a few months ago, on https://src.fedoraproject.org/rpms/prosody/c/33be1282d450ee0ab1032fa91412bcf747c23ad7?branch=rawhide
Proposed patch, switch back to libidn: https://src.fedoraproject.org/rpms/prosody/pull-request/1
As mentioned there (https://src.fedoraproject.org/rpms/prosody/pull-request/1#comment-73097), this is not a good idea. The only issue here is libicu, which was unfortunately broken from libicu-67.1-5.fc34 to libicu-67.1-6.fc34: $ cat /tmp/e.lua package.path = '/usr/lib64/prosody/?.lua;'; package.cpath = '/usr/lib64/prosody/?.so;'; local encodings = require "util.encodings"; $ $ rpm -q libicu libicu-67.1-6.fc34.x86_64 $ $ lua /tmp/e.lua terminate called after throwing an instance of 'std::system_error' what(): Unknown error -1 Aborted (core dumped) $ $ dnf install https://kojipkgs.fedoraproject.org//packages/icu/67.1/5.fc34/x86_64/libicu-67.1-5.fc34.x86_64.rpm -y >> /dev/null 2>&1 $ $ rpm -q libicu libicu-67.1-5.fc34.x86_64 $ $ lua /tmp/e.lua $
I tried a rebuild of the icu package: https://koji.fedoraproject.org/koji/taskinfo?taskID=66854119 (in case this was some gcc errors or something) No luck, still fail.
Yes, I've tried that last night as well. Looks like the build root difference between libicu-67.1-5.fc34 and libicu-67.1-6.fc34 causes the impact.
So, to rule out gcc upgrade, I built icu using clang: https://koji.fedoraproject.org/koji/taskinfo?taskID=66916464 And the error is still here (but maybe I did the gcc to clang switch wrong).
Seeing that it crashes immediately under invocation from Lua, #6 0x00007f0e501a2830 _ZSt20__throw_system_errori (libstdc++.so.6 + 0xa4830) #7 0x00007f0e51e9efff _ZN6icu_6720umtx_initImplPreInitERNS_9UInitOnceE (libicuuc.so.67 + 0x68fff) #8 0x00007f0e51f6c8ff usprep_open_67 (libicuuc.so.67 + 0x1368ff) #9 0x00007f0e52022f8a init_icu (encodings.so + 0x1f8a) #10 0x00007f0e52023051 luaopen_util_encodings (encodings.so + 0x2051) and Lua wasn't rebuilt since the mass rebuild on 2021-01-26 (i.e. not "Rebuilt for removed libstdc++ symbol (#1937698)" which is the only change between icu-67.1-5.fc34 and icu-67.1-6.fc34), does this still happen with lua-5.4.3-1.fc34 which was built just yesterday? See https://koji.fedoraproject.org/koji/buildinfo?buildID=1741480
Ah darn, luaopen_util_encodings() and init_icu() are not from Lua but from Prosody ... which also wasn't rebuilt after 2021-02-26 (may or may not be related). Odd enough, init_icu() does not call usprep_open() but usprep_openByType(), which in turn calls usprep_open(). However, umtx_initImplPreInit() is called by a umtx_initOnce() template, which in usprep context is only used in a static initCache() function called by usprep_getProfile() called by usprep_open(). Those two are missing from the stack trace, initCache() may have been inlined but I doubt usprep_getProfile() was. Anyway, mutex umtx_initImplPreInit() is used by umtx_initOnce() in 100 different places, ICU is a critical path library and I doubt it's a general failure. I'd recommend to try if a rebuild of Prosody solves the problem.
As per comment #8, a simple rebuild of Prosody with lua-5.4.2-2.fc34 in buildroot does not help. But let's see if this changes with lua-5.4.3-1.fc34 in buildroot (as suggested in comment #15).
No, it doesn't help: https://koji.fedoraproject.org/koji/taskinfo?taskID=66931755
One of the Prosody upstream developers had a look to this today. As of writing, it seems like the issue can be alternatively triggered in Fedora 34+ via `-Wl,--as-needed`: Fedora 34 RPM, segfaulting (ldd /usr/lib64/prosody/util/encodings.so): linux-vdso.so.1 (0x00007fffb9f9b000) libicuuc.so.67 => /lib64/libicuuc.so.67 (0x00007f4321480000) libc.so.6 => /lib64/libc.so.6 (0x00007f43212b0000) libicudata.so.67 => /lib64/libicudata.so.67 (0x00007f431f790000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f431f788000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f431f568000) libm.so.6 => /lib64/libm.so.6 (0x00007f431f420000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f431f400000) /lib64/ld-linux-x86-64.so.2 (0x00007f4321680000) Fedora 34 without `-Wl,--as-needed`, working (ldd encodings.so): linux-vdso.so.1 (0x00007fff183db000) libicui18n.so.67 => /lib64/libicui18n.so.67 (0x00007fc96de98000) libicudata.so.67 => /lib64/libicudata.so.67 (0x00007fc96c378000) libicuuc.so.67 => /lib64/libicuuc.so.67 (0x00007fc96c188000) libc.so.6 => /lib64/libc.so.6 (0x00007fc96bfb8000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc96bf90000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fc96bd70000) libm.so.6 => /lib64/libm.so.6 (0x00007fc96bc28000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc96bc08000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fc96bc00000) /lib64/ld-linux-x86-64.so.2 (0x00007fc96e1b0000) Difference: libicui18n.so.6? and libpthread.so.0 are linked in the working build Debian 10 without `-Wl,--as-needed`, working (ldd encodings.so): linux-vdso.so.1 (0x00007fff739b0000) libicuuc.so.63 => /usr/lib/x86_64-linux-gnu/libicuuc.so.63 (0x00007fca2a2e5000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fca2a124000) libicudata.so.63 => /usr/lib/x86_64-linux-gnu/libicudata.so.63 (0x00007fca28734000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fca28713000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fca2870e000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fca2858a000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fca28405000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fca283eb000) /lib64/ld-linux-x86-64.so.2 (0x00007fca2a4df000) Difference: libicui18n.so.6? is absent and libpthread.so.0 is linked in the working build
FEDORA-2021-aef3ad6c18 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-aef3ad6c18
Sorry, "Debian 10 without `-Wl,--as-needed`, working (ldd encodings.so)" is wrong, it should have been "Debian 10 WITH `-Wl,--as-needed`, working (ldd encodings.so)" instead.
FEDORA-2021-aef3ad6c18 has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-aef3ad6c18` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-aef3ad6c18 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Great finding, thanks!
But I'm not sure whether this is some GCC/linker issue, because -lpthread shouldn't be removed by `-Wl,--as-needed` (the -lpthread is IMHO introduced and required by libicu). And it's still interesting that it happens since a rebuild of libicu.
There was almost no ABI change with the rebuild: $ abipkgdiff libicu-67.1-5.fc34.x86_64.rpm libicu-67.1-6.fc34.x86_64.rpm ================ changes of 'libicuuc.so.67.1'=============== Functions changes summary: 0 Removed, 0 Changed, 0 Added function Variables changes summary: 0 Removed, 0 Changed, 0 Added variable Function symbols changes summary: 0 Removed, 1 Added function symbol not referenced by debug info Variable symbols changes summary: 0 Removed, 0 Added variable symbol not referenced by debug info 1 Added function symbol not referenced by debug info: [A] _ZZNSt9once_flag18_Prepare_executionC4IZSt9call_onceIRFvvEJEEvRS_OT_DpOT0_EUlvE_EERS6_ENUlvE_4_FUNEv ================ end of changes of 'libicuuc.so.67.1'=============== Given that's related to std::once_flag, could it be also interfering with umtx_initOnce ?
FEDORA-2021-aef3ad6c18 has been pushed to the Fedora 34 stable repository. If problem still persists, please make note of it in this bug report.
This message is a reminder that Fedora Linux 34 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 34 on 2022-06-07. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '34'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 34 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Fedora Linux 34 entered end-of-life (EOL) status on 2022-06-07. Fedora Linux 34 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. Thank you for reporting this bug and we are sorry it could not be fixed.