Bug 1954178 - Prosody fails to start (due to libicu)
Summary: Prosody fails to start (due to libicu)
Keywords:
Status: ON_QA
Alias: None
Product: Fedora
Classification: Fedora
Component: icu
Version: 34
Hardware: x86_64
OS: Linux
high
unspecified
Target Milestone: ---
Assignee: Eike Rathke
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-27 18:07 UTC by Michael S.
Modified: 2021-05-01 02:57 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)

Description Michael S. 2021-04-27 18:07:45 UTC
Description of problem:

prosody fail to start after a upgrade to F34


Version-Release number of selected component (if applicable):
# rpm -q prosody lua
prosody-0.11.8-2.fc34.x86_64
lua-5.4.2-2.fc34.x86_64

How reproducible:

each time (on my instance)

Steps to Reproduce:
1. try to start prosody
2.
3.

Actual results:
Apr 27 20:05:17 karona prosody[2514]: terminate called after throwing an instance of 'std::system_error'
Apr 27 20:05:17 karona prosody[2514]:   what():  Unknown error -1


Expected results:

no error

Additional info:

I have a complete traceback. I will add comment as I find more information.

Comment 1 Michael S. 2021-04-27 18:09:30 UTC
Apr 27 20:05:18 karona systemd-coredump[2516]: [🡕] Process 2514 (prosody) of user 993 dumped core.
                                               
                                               Stack trace of thread 2514:
                                               #0  0x00007f0e521e2292 raise (libc.so.6 + 0x3d292)
                                               #1  0x00007f0e521cb8a4 abort (libc.so.6 + 0x268a4)
                                               #2  0x00007f0e5019fa46 _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold (libstdc++.so.6 + 0xa1a46)
                                               #3  0x00007f0e501ab28c _ZN10__cxxabiv111__terminateEPFvvE (libstdc++.so.6 + 0xad28c)
                                               #4  0x00007f0e501ab2f7 _ZSt9terminatev (libstdc++.so.6 + 0xad2f7)
                                               #5  0x00007f0e501ab599 __cxa_throw (libstdc++.so.6 + 0xad599)
                                               #6  0x00007f0e501a2830 _ZSt20__throw_system_errori (libstdc++.so.6 + 0xa4830)
                                               #7  0x00007f0e51e9efff _ZN6icu_6720umtx_initImplPreInitERNS_9UInitOnceE (libicuuc.so.67 + 0x68fff)
                                               #8  0x00007f0e51f6c8ff usprep_open_67 (libicuuc.so.67 + 0x1368ff)
                                               #9  0x00007f0e52022f8a init_icu (encodings.so + 0x1f8a)
                                               #10 0x00007f0e52023051 luaopen_util_encodings (encodings.so + 0x2051)
                                               #11 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3)
                                               #12 0x00007f0e523e352b ccall.lto_priv.0 (liblua-5.4.so + 0x1752b)
                                               #13 0x00007f0e523d90a4 lua_callk (liblua-5.4.so + 0xd0a4)
                                               #14 0x00007f0e523f466b ll_require (liblua-5.4.so + 0x2866b)
                                               #15 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3)
                                               #16 0x00007f0e523dd82d f_call (liblua-5.4.so + 0x1182d)
                                               #17 0x00007f0e523e2853 luaD_rawrunprotected (liblua-5.4.so + 0x16853)
                                               #18 0x00007f0e523e5022 luaD_pcall (liblua-5.4.so + 0x19022)
                                               #19 0x00007f0e523d921c lua_pcallk (liblua-5.4.so + 0xd21c)
                                               #20 0x00007f0e523dec45 luaB_pcall (liblua-5.4.so + 0x12c45)
                                               #21 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3)
                                               #22 0x00007f0e523f97ed luaV_execute (liblua-5.4.so + 0x2d7ed)
                                               #23 0x00007f0e523e3544 ccall.lto_priv.0 (liblua-5.4.so + 0x17544)
                                               #24 0x00007f0e523d90a4 lua_callk (liblua-5.4.so + 0xd0a4)
                                               #25 0x00007f0e523f466b ll_require (liblua-5.4.so + 0x2866b)
                                               #26 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3)
                                               #27 0x00007f0e523f97ed luaV_execute (liblua-5.4.so + 0x2d7ed)
                                               #28 0x00007f0e523e3544 ccall.lto_priv.0 (liblua-5.4.so + 0x17544)
                                               #29 0x00007f0e523d90a4 lua_callk (liblua-5.4.so + 0xd0a4)
                                               #30 0x00007f0e523f466b ll_require (liblua-5.4.so + 0x2866b)
                                               #31 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3)
                                               #32 0x00007f0e523f97ed luaV_execute (liblua-5.4.so + 0x2d7ed)
                                               #33 0x00007f0e523dd846 f_call (liblua-5.4.so + 0x11846)
                                               #34 0x00007f0e523e2853 luaD_rawrunprotected (liblua-5.4.so + 0x16853)
                                               #35 0x00007f0e523e5022 luaD_pcall (liblua-5.4.so + 0x19022)
                                               #36 0x00007f0e523d921c lua_pcallk (liblua-5.4.so + 0xd21c)
                                               #37 0x000055716f6f4a4f docall (lua + 0x2a4f)
                                               #38 0x000055716f6f5607 pmain (lua + 0x3607)
                                               #39 0x00007f0e523e33d3 luaD_precall (liblua-5.4.so + 0x173d3)
                                               #40 0x00007f0e523dd82d f_call (liblua-5.4.so + 0x1182d)
                                               #41 0x00007f0e523e2853 luaD_rawrunprotected (liblua-5.4.so + 0x16853)
                                               #42 0x00007f0e523e5022 luaD_pcall (liblua-5.4.so + 0x19022)
                                               #43 0x00007f0e523d921c lua_pcallk (liblua-5.4.so + 0xd21c)
                                               #44 0x000055716f6f470f main (lua + 0x270f)
                                               #45 0x00007f0e521ccb75 __libc_start_main (libc.so.6 + 0x27b75)
                                               #46 0x000055716f6f47be _start (lua + 0x27be)


So this is a issue with icu_67::umtx_initImplPreInit, it seems.

And prosodyctl also fail:
# prosodyctl 
terminate called after throwing an instance of 'std::system_error'
  what():  Unknown error -1
Aborted (core dumped)

Comment 2 Michael S. 2021-04-27 18:22:18 UTC
So somewhere here:

https://hg.prosody.im/trunk/file/tip/util-src/encodings.c#l342

Comment 3 Michael S. 2021-04-27 18:23:47 UTC
So prosodyctl work on a fresh installation. This is quite curious.

Comment 4 Michael S. 2021-04-27 18:36:55 UTC
So to be precise, the line that fail is https://hg.prosody.im/trunk/file/tip/util-src/encodings.c#l345

#11 usprep_getProfile (status=0x7fffffffd7a4, name=0x7ffff7b7c40c "rfc3491", path=0x0) at /usr/src/debug/icu-67.1-6.fc34.x86_64/source/common/usprep.cpp:311
#12 usprep_open_67 (path=0x0, name=0x7ffff7b7c40c "rfc3491", status=0x7fffffffd7a4) at /usr/src/debug/icu-67.1-6.fc34.x86_64/source/common/usprep.cpp:401

Comment 5 Robert Scheck 2021-04-27 18:39:30 UTC
Given prosody and prosodyctl have both been working fine while the package was built [1][2], prosody might be the victim of later introduced breaking changes in Fedora (maybe Lua or ICU)...assuming that you are talking about an empty, unconfigured prosody here.

[1] https://kojipkgs.fedoraproject.org//packages/prosody/0.11.8/2.fc34/data/logs/x86_64/build.log
[2] https://src.fedoraproject.org/rpms/prosody/c/9354a84e62995723d190b95b85072541b26f3a6f?branch=f34

Comment 6 Michael S. 2021-04-27 19:04:55 UTC
Ok so #3 was wrong, I forgot I didn't upgrade this specific server.

On F34, the shortest reproducer I have is the following:

# cat /tmp/e.lua 
package.path =  '/usr/lib64/prosody/?.lua;'; 
package.cpath = '/usr/lib64/prosody/?.so;';
local encodings = require "util.encodings";


# lua /tmp/e.lua
terminate called after throwing an instance of 'std::system_error'
  what():  Unknown error -1
Aborted (core dumped)

Running gdb show something in libicu fail.

Comment 7 Michael S. 2021-04-27 19:14:53 UTC
I am trying a scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=66813543

Comment 8 Michael S. 2021-04-27 19:20:19 UTC
Fail as well: https://koji.fedoraproject.org/koji/getfile?taskID=66813654&volume=DEFAULT&name=build.log&offset=-4000

++ pwd
+ export PROSODY_DATADIR=/builddir/build/BUILD/prosody-0.11.8/tests/data
+ PROSODY_DATADIR=/builddir/build/BUILD/prosody-0.11.8/tests/data
+ ./prosodyctl about
terminate called after throwing an instance of 'std::system_error'
  what():  Unknown error -1
/var/tmp/rpm-tmp.V9fT0d: line 47: 55130 Aborted                 (core dumped) ./prosodyctl about

Comment 9 Michael S. 2021-04-27 22:39:42 UTC
Switching from icu to idn fixed the problem:
https://koji.fedoraproject.org/koji/taskinfo?taskID=66824480

That's a workaround, and to be honest, I do not know the impact. 
Using ICU was added a few months ago, on https://src.fedoraproject.org/rpms/prosody/c/33be1282d450ee0ab1032fa91412bcf747c23ad7?branch=rawhide

Comment 10 Michael S. 2021-04-27 23:26:35 UTC
Proposed patch, switch back to libidn:

https://src.fedoraproject.org/rpms/prosody/pull-request/1

Comment 11 Robert Scheck 2021-04-27 23:37:04 UTC
As mentioned there (https://src.fedoraproject.org/rpms/prosody/pull-request/1#comment-73097), this is not a good idea.

The only issue here is libicu, which was unfortunately broken from libicu-67.1-5.fc34 to libicu-67.1-6.fc34:


$ cat /tmp/e.lua 
package.path =  '/usr/lib64/prosody/?.lua;'; 
package.cpath = '/usr/lib64/prosody/?.so;';
local encodings = require "util.encodings";
$ 

$ rpm -q libicu
libicu-67.1-6.fc34.x86_64
$

$ lua /tmp/e.lua
terminate called after throwing an instance of 'std::system_error'
  what():  Unknown error -1
Aborted (core dumped)
$

$ dnf install https://kojipkgs.fedoraproject.org//packages/icu/67.1/5.fc34/x86_64/libicu-67.1-5.fc34.x86_64.rpm -y >> /dev/null 2>&1
$ 

$ rpm -q libicu
libicu-67.1-5.fc34.x86_64
$

$ lua /tmp/e.lua
$

Comment 12 Michael S. 2021-04-28 11:02:04 UTC
I tried a rebuild of the icu package: https://koji.fedoraproject.org/koji/taskinfo?taskID=66854119
(in case this was some gcc errors or something)

No luck, still fail.

Comment 13 Robert Scheck 2021-04-28 13:24:55 UTC
Yes, I've tried that last night as well. Looks like the build root difference between libicu-67.1-5.fc34 and libicu-67.1-6.fc34 causes the impact.

Comment 14 Michael S. 2021-04-29 10:36:56 UTC
So, to rule out gcc upgrade, I built icu using clang: https://koji.fedoraproject.org/koji/taskinfo?taskID=66916464 

And the error is still here (but maybe I did the gcc to clang switch wrong).

Comment 15 Eike Rathke 2021-04-29 13:45:08 UTC
Seeing that it crashes immediately under invocation from Lua,

#6  0x00007f0e501a2830 _ZSt20__throw_system_errori (libstdc++.so.6 + 0xa4830)
#7  0x00007f0e51e9efff _ZN6icu_6720umtx_initImplPreInitERNS_9UInitOnceE (libicuuc.so.67 + 0x68fff)
#8  0x00007f0e51f6c8ff usprep_open_67 (libicuuc.so.67 + 0x1368ff)
#9  0x00007f0e52022f8a init_icu (encodings.so + 0x1f8a)
#10 0x00007f0e52023051 luaopen_util_encodings (encodings.so + 0x2051)

and Lua wasn't rebuilt since the mass rebuild on 2021-01-26 (i.e. not "Rebuilt for removed libstdc++ symbol (#1937698)" which is the only change between icu-67.1-5.fc34 and icu-67.1-6.fc34), does this still happen with lua-5.4.3-1.fc34 which was built just yesterday?
See https://koji.fedoraproject.org/koji/buildinfo?buildID=1741480

Comment 16 Eike Rathke 2021-04-29 14:53:06 UTC
Ah darn, luaopen_util_encodings() and init_icu() are not from Lua but from Prosody ... which also wasn't rebuilt after 2021-02-26 (may or may not be related).
Odd enough, init_icu() does not call usprep_open() but usprep_openByType(), which in turn calls usprep_open(). However, umtx_initImplPreInit() is called by a umtx_initOnce() template, which in usprep context is only used in a static initCache() function called by usprep_getProfile() called by usprep_open(). Those two are missing from the stack trace, initCache() may have been inlined but I doubt usprep_getProfile() was. Anyway, mutex umtx_initImplPreInit() is used by umtx_initOnce() in 100 different places, ICU is a critical path library and I doubt it's a general failure. I'd recommend to try if a rebuild of Prosody solves the problem.

Comment 17 Robert Scheck 2021-04-29 16:14:34 UTC
As per comment #8, a simple rebuild of Prosody with lua-5.4.2-2.fc34 in buildroot does not help. But let's see if this changes with lua-5.4.3-1.fc34 in buildroot (as suggested in comment #15).

Comment 18 Robert Scheck 2021-04-29 16:22:48 UTC
No, it doesn't help: https://koji.fedoraproject.org/koji/taskinfo?taskID=66931755

Comment 19 Robert Scheck 2021-04-29 23:25:34 UTC
One of the Prosody upstream developers had a look to this today. As of writing, it seems like the issue can be alternatively triggered in Fedora 34+ via `-Wl,--as-needed`:

Fedora 34 RPM, segfaulting (ldd /usr/lib64/prosody/util/encodings.so):
linux-vdso.so.1 (0x00007fffb9f9b000)
libicuuc.so.67 => /lib64/libicuuc.so.67 (0x00007f4321480000)
libc.so.6 => /lib64/libc.so.6 (0x00007f43212b0000)
libicudata.so.67 => /lib64/libicudata.so.67 (0x00007f431f790000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f431f788000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f431f568000)
libm.so.6 => /lib64/libm.so.6 (0x00007f431f420000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f431f400000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4321680000)

Fedora 34 without `-Wl,--as-needed`, working (ldd encodings.so):
linux-vdso.so.1 (0x00007fff183db000)
libicui18n.so.67 => /lib64/libicui18n.so.67 (0x00007fc96de98000)
libicudata.so.67 => /lib64/libicudata.so.67 (0x00007fc96c378000)
libicuuc.so.67 => /lib64/libicuuc.so.67 (0x00007fc96c188000)
libc.so.6 => /lib64/libc.so.6 (0x00007fc96bfb8000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc96bf90000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fc96bd70000)
libm.so.6 => /lib64/libm.so.6 (0x00007fc96bc28000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc96bc08000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fc96bc00000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc96e1b0000)

Difference: libicui18n.so.6? and libpthread.so.0 are linked in the working build

Debian 10 without `-Wl,--as-needed`, working (ldd encodings.so):
linux-vdso.so.1 (0x00007fff739b0000)
libicuuc.so.63 => /usr/lib/x86_64-linux-gnu/libicuuc.so.63 (0x00007fca2a2e5000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fca2a124000)
libicudata.so.63 => /usr/lib/x86_64-linux-gnu/libicudata.so.63 (0x00007fca28734000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fca28713000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fca2870e000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fca2858a000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fca28405000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fca283eb000)
/lib64/ld-linux-x86-64.so.2 (0x00007fca2a4df000)

Difference: libicui18n.so.6? is absent and libpthread.so.0 is linked in the working build

Comment 20 Fedora Update System 2021-04-30 00:12:30 UTC
FEDORA-2021-aef3ad6c18 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-aef3ad6c18

Comment 21 Robert Scheck 2021-04-30 00:15:23 UTC
Sorry, "Debian 10 without `-Wl,--as-needed`, working (ldd encodings.so)" is wrong, it should have been "Debian 10 WITH `-Wl,--as-needed`, working (ldd encodings.so)" instead.

Comment 22 Fedora Update System 2021-04-30 01:43:47 UTC
FEDORA-2021-aef3ad6c18 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-aef3ad6c18`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-aef3ad6c18

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 23 Eike Rathke 2021-04-30 09:57:49 UTC
Great finding, thanks!

Comment 24 Robert Scheck 2021-04-30 10:01:55 UTC
But I'm not sure whether this is some GCC/linker issue, because -lpthread shouldn't be removed by `-Wl,--as-needed` (the -lpthread is IMHO introduced and required by libicu). And it's still interesting that it happens since a rebuild of libicu.

Comment 25 Michael S. 2021-04-30 14:14:31 UTC
There was almost no ABI change with the rebuild:

$ abipkgdiff libicu-67.1-5.fc34.x86_64.rpm libicu-67.1-6.fc34.x86_64.rpm 
================ changes of 'libicuuc.so.67.1'===============
  Functions changes summary: 0 Removed, 0 Changed, 0 Added function
  Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
  Function symbols changes summary: 0 Removed, 1 Added function symbol not referenced by debug info
  Variable symbols changes summary: 0 Removed, 0 Added variable symbol not referenced by debug info

  1 Added function symbol not referenced by debug info:

    [A] _ZZNSt9once_flag18_Prepare_executionC4IZSt9call_onceIRFvvEJEEvRS_OT_DpOT0_EUlvE_EERS6_ENUlvE_4_FUNEv

================ end of changes of 'libicuuc.so.67.1'===============

Given that's related to std::once_flag, could it be also interfering with umtx_initOnce ?

Comment 26 Fedora Update System 2021-05-01 02:57:50 UTC
FEDORA-2021-aef3ad6c18 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.