Bug 2012249
| Summary: | gnutls_priority_set_direct occasionally fails with "The request is invalid" | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Richard W.M. Jones <rjones> | ||||||||||
| Component: | gnutls | Assignee: | Daiki Ueno <dueno> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Alexander Sosedkin <asosedki> | ||||||||||
| Severity: | high | Docs Contact: | |||||||||||
| Priority: | high | ||||||||||||
| Version: | 9.0 | CC: | asosedki, berrange, eblake, jeckersb, kkiwi, michele, ssorce | ||||||||||
| Target Milestone: | rc | Keywords: | Triaged | ||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | gnutls-3.7.2-9.el9 | Doc Type: | No Doc Update | ||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2022-05-17 15:52:13 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Attachments: |
|
||||||||||||
|
Description
Richard W.M. Jones
2021-10-08 16:22:26 UTC
Created attachment 1830898 [details]
log file without gnutls debugging
Only seems to be reproducible in RHEL 9. I cannot reproduce it in Fedora. Some think it might be connected to this non-upstream change which is only in RHEL 9: https://gitlab.com/gnutls/gnutls/-/merge_requests/1427 3.7.2-4 is the package that re-introduced LTO enablement after a long time. As it created several obscure issues in tests when running on aarch64 and ppc64le, we disabled LTO on those arches in 3.7.2-6. As far as I read from the original thread, the failure seems to happen only on aarch64 with 3.7.2-4, so I would suggest building with the latest gnutls package (3.7.2-7). Created attachment 1836609 [details]
synch-parallel-tls.sh.log
My locally testing is on x86-64.
The bug still happens (perhaps less often?) with gnutls-3.7.2-7.el9.x86_64
Attached latest log of the failure.
(In reply to Daiki Ueno from comment #3) > 3.7.2-4 is the package that re-introduced LTO enablement after a long time. > As it created several obscure issues in tests when running on aarch64 and > ppc64le, we disabled LTO on those arches in 3.7.2-6. (In reply to Richard W.M. Jones from comment #4) > My locally testing is on x86-64. > > The bug still happens (perhaps less often?) with gnutls-3.7.2-7.el9.x86_64 Perhaps worth doing a gnutls scratch build with LTO disabled on x86_64 too, and seeing if that solves it, as LTO has been a source of many wierd non-deterministic bugs. Oh interesting, I thought LTO had been disabled on all architectures. I did a scratch build with LTO disabled on x86-64 too which I will test once it has finished: https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=746280 (In reply to Richard W.M. Jones from comment #6) > https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=746280 This did *not* fix the problem, so it's not LTO. (In reply to Richard W.M. Jones from comment #7) > (In reply to Richard W.M. Jones from comment #6) > > https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=746280 > > This did *not* fix the problem, so it's not LTO. My apologies, I was reading the wrong log file. In fact this package does fix the problem, so it is a problem related to LTO on x86-64. Oh I hate intermittent errors! Just as I hit submit on that comment, the test which had run successfully for 100+ cycles failed again with the same problem. This is with LTO disabled, so the problem still seems to be present and NOT related to LTO after all. I'm going to try this with upstream gnutls, and also see if I can get a more reliable test case. It would be really nice if gnutls got rid of the requirement for autogen. This is not available on RHEL 9 and almost impossible to build on RHEL 9 because it depends on both itself and gnulib. Created attachment 1836943 [details]
Log with GNUTLS_DEBUG_LEVEL=10
(In reply to Richard W.M. Jones from comment #11) > Created attachment 1836943 [details] > Log with GNUTLS_DEBUG_LEVEL=10 OK, thank you so much for looking into this; it seems indeed like a race condition: the resolved priority string is stored in the global variable system_wide_priority_string, while the other threads may independently update the variable, without lock. A similar race seems to be found in _gnutls_unload_system_priorities() for system_wide_priority_strings, though it is not called so frequently as the caller checks mtime of the config file. I'll create a patch shortly. Created attachment 1836965 [details]
tlsthread.c
This is a reproducer. It fails for me reliably and in a couple
of interesting ways. It does not require anything except RHEL 9
and gnutls-devel.
$ gcc -O2 -Wall -pthread tlsthread.c -o tlsthread -lgnutls
$ while ./tlsthread ; do echo -n . ; done
.............................................................................................................................................../tlsthread: gnutls_priority_set_direct: The request is invalid.
Sometimes it fails indicating memory corruption:
$ while ./tlsthread ; do echo -n .; done
tcache_thread_shutdown(): unaligned tcache chunk detected
Aborted (core dumped)
(The stack trace from this was not very interesting)
Thank you for the reproducer. I've created a scratch build with the proposed fix: https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=748121 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (new packages: gnutls), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:3937 |