Fedora Account System
Red Hat Associate
Red Hat Customer
Since around 2026-08-13, in Fedora Rawhide, we've been seeing intermittent failures of the openQA desktop_update_graphical test. This test tests graphical installation of system updates. On GNOME, it uses GNOME Software, and is working reliably. On KDE, it uses Discover, and frequently the test fails because Discover never successfully completes startup. I've managed to get openQA to attach gdb to the stuck process and produce a backtrace, and from the backtrace it appears to be stuck in OPENSSL_init_crypto: #0 futex_wait (futex_word=0x7f349a95b810 <config>, expected=1, private=0) at ../sysdeps/nptl/futex-internal.h:126 __ret = -512 err = <optimized out> #1 futex_wait_simple (futex_word=0x7f349a95b810 <config>, expected=1, private=0) at ../sysdeps/nptl/futex-internal.h:157 __ignored_value = <optimized out> #2 __pthread_once_slow (once_control=0x7f349a95b810 <config>, init_routine=0x7f349a69bef0 <ossl_init_config_ossl_>) at pthread_once.c:105 val = 1 newval = 1 #3 0x00007f349c4f3019 in ___pthread_once (once_control=<optimized out>, init_routine=<optimized out>) at pthread_once.c:143 val = <optimized out> #4 0x00007f349a6a7d7d in CRYPTO_THREAD_run_once (once=<optimized out>, init=<optimized out>) at crypto/threads_pthread.c:1017 No locals. #5 0x00007f349a69c5d9 in OPENSSL_init_crypto (opts=64, settings=0x0) at crypto/init.c:463 ret = <optimized out> loading = <optimized out> tmp = 2 aloaddone = 1 __func__ = "OPENSSL_init_crypto" #6 0x00007f349a6acabf in ossl_provider_find (libctx=libctx@entry=0x0, name=name@entry=0x7f349cabbf7a "default", noconfig=noconfig@entry=0) at crypto/provider_core.c:418 tmpl = {flag_initialized = 0, flag_activated = 0, flag_lock = 0x0, refcnt = {val = 0}, activatecnt_lock = 0x0, activatecnt = 0, name = 0x0, path = 0x0, module = 0x0, init_function = 0x0, parameters = 0x0, libctx = 0x0, store = 0x0, error_lib = 0, error_strings = 0x0, teardown = 0x0, gettable_params = 0x0, get_params = 0x0, get_capabilities = 0x0, self_test = 0x0, random_bytes = 0x0, query_operation = 0x0, unquery_operation = 0x0, operation_bits = 0x0, operation_bits_sz = 0, opbits_lock = 0x0, handle = 0x0, ischild = 0, provctx = 0x0, dispatch = 0x0} i = <optimized out> store = 0x7f345c0095d0 prov = 0x0 #7 0x00007f349a6ad308 in OSSL_PROVIDER_try_load_ex (libctx=0x0, name=0x7f349cabbf7a "default", params=0x0, retain_fallbacks=1) at crypto/provider.c:25 prov = <optimized out> actual = 0x3ec0 isnew = 0 ... On the theory that this may be due to entropy starvation, I tried having the test install and enable haveged before running Discover, but this does not appear to have helped. The bug has several annoying properties. It seems to go in 'waves' - almost every test will fail for a while, then all tests will pass for a while, then it'll go back to failing. I ran the test well over 100 times yesterday afternoon and evening (my time) without a single failure; I then ran it multiple times this morning / early afternoon and nearly every attempt failed. It doesn't seem to be reproducible locally, at least several of us have tried and not once got it to happen in a local VM. There's no obvious suspect. The bug doesn't appear to affect Fedora 45 tests, at least not yet. I went looking for things that changed in Rawhide but not Fedora 45 around August 13 and did not find anything that looks like it could have caused this. openssl has not changed since July and is the same in F45 and Rawhide. plasma-discover similarly doesn't fit the timeframe. qt6 doesn't either. Filing this on openssl nevertheless because I figured at least someone might have some idea what's going on, or a way to debug it further. The 'entropy starvation' idea was my best guess, but...I'd really have thought haveged would solve it if that was the case.
Created attachment 2155735 [details] full backtrace
Is this a recent issue? And if so, is pkcs11-provider installed by chance?
Yeah, as I said, it seems to have started since 2026-08-13. pkcs11-provider almost certainly is installed, yeah (it seems to be in a stock KDE install).
Oooh. I see where you're coming from. https://koji.fedoraproject.org/koji/buildinfo?buildID=3077950 fits the timeline (it was tagged right on August 13) and is Rawhide-only. And the change sounds interesting: "- Revert "Temporarily disable loading by default"". Should I maybe try a scratch build with that flipped back and see if the problem goes away?
...or I suppose I could just hack up the test to remove the package, that'd be an easier test...
Aha, I think you may be onto something. I got 37 passes with a `dnf -y remove pkcs11-provider` added to the test, then took it out and re-ran, and immediately got several failures.
Thank you for the report! The change was explicitly made only for rawhide to catch possible issues early without breaking Fedora 45, but our understanding was the issues we had when we tried to do this first time were resolved in openssl already. While the pkcs11 provider change is likely triggering the issue, the traces do not show any trace of pkcs11 provider and all point to the openssl code or qtbase. Reading through the code calling this openssl function OSSL_PROVIDER_try_load_ex, if I see, right, its called repetitively on initialization of the hash mechanism (and possibly from different threads?), instead of calling this once during initialization, which might or might not be an issue. Adam, do you know if there are some threads involved when the process hangs? https://code.qt.io/cgit/qt/qtbase.git/tree/src/corelib/tools/qcryptographichash.cpp#n567 The part of openssl trace suggests just that this is the generic library init function waiting on `pthread_once` and if I see right, its the configuration file processing: https://github.com/openssl/openssl/blob/1e963a8680ec78ad2072792c7a1a71f3c530bd2e/crypto/init.c#L463 This might point to the pkcs11 provider indirectly (as its included in the configuration file), but from what I see it points out to some (thread?) synchronization during openssl initialization, rather than pkcs11-provider issue itself. I think it will require some more investigation from OpenSSL developers.
It is a threaded app, yeah. I couldn't actually get a full threaded backtrace because the VM didn't have enough RAM and gdb would get OOM-killed trying to generate it! I can set things back to reproduce the problem and hack up one run of the test to have more RAM and take another swing at generating it today, since it seems like it may be important.
Created attachment 2155841 [details] threaded backtrace
Thank you! The Thread 2 (Thread 0x7f128465e6c0 (LWP 2336) "FlatpakRefreshA") is stuck inside of the OPENSSL_init_crypto() which is loading the pkcs11 provider and hangs on dlopen lock (I think without executing any pkcs11 provider code so far). This is called from `curl_easy_init()` from within some flatpak handling code downloading stuff. The Thread 1 (Thread 0x7f12b5aa4f80 (LWP 2332) "plasma-discover") afterward waits for a lock in OPENSSL_init_crypto(), which comes from the `QCryptographicHashPrivate::EVP::EVP` initialization which I mentioned in the previous comment and is likely waiting for the Thread 2 to finish. It is not yet completely clear to me why the lock in dlopen keeps waiting (and on what does it wait). Other threads seems to be waiting on poll, which is expected.
FEDORA-2026-8ae26b4e2f (openssl-4.0.2-3.fc46) has been submitted as an update to Fedora 46. https://bodhi.fedoraproject.org/updates/FEDORA-2026-8ae26b4e2f
Adam, could you please check with openssl-4.0.2-3.fc46? I've backported a patch which may be relevant (the race that it fixes was causing other issues when loading providers in threaded openssl applications). I'm not completely sure whether it'll help, but it may be worth checking.
FEDORA-2026-8ae26b4e2f (openssl-4.0.2-3.fc46) has been pushed to the Fedora 46 stable repository. If problem still persists, please make note of it in this bug report.