RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1727865 - Initialization of cyrus-sasl library is not thread safe causing libvirt client crashes
Summary: Initialization of cyrus-sasl library is not thread safe causing libvirt clien...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: libvirt
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Daniel Berrangé
QA Contact: yafu
URL:
Whiteboard:
Depends On:
Blocks: 1727866
TreeView+ depends on / blocked
 
Reported: 2019-07-08 11:56 UTC by Daniel Berrangé
Modified: 2020-11-04 02:54 UTC (History)
3 users (show)

Fixed In Version: libvirt-6.0.0-17.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1727866 (view as bug list)
Environment:
Last Closed: 2020-11-04 02:53:02 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
SASL race demo program (1.23 KB, text/x-csrc)
2019-07-08 11:56 UTC, Daniel Berrangé 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2020:4676 0 None None None 2020-11-04 02:53:55 UTC

Description Daniel Berrangé 2019-07-08 11:56:53 UTC 
Created attachment 1588340 [details]
SASL race demo program

Description of problem:
The sasl_client_init and/or sasl_server_init methods are required in order to initialize the SASL library global state.

Libvirt currently assumes the methods are no-ops if called more than once which is only true if the calls are serialized. The protection against multiple invocation uses a simple integer counter with no atomic access/increments, nor is any libpthread protection used. This is a clear race for threads.

Version-Release number of selected component (if applicable):
libvirt-5.0.0-11.el8

How reproducible:
Fairly often in a threaded test case

Steps to Reproduce:
1. Edit /etc/libvirt/libvirt.conf to set

   auth_unix_rw = "sasl"

2. Edit /etc/sasl2/libvirt.conf to set

   mech-list: scram-sha-1

3. As root run   "saslpasswd2 -a libvirt test"  and enter "123456" as password when prompted

4. systemctl restart libvirtd

5. virsh -c qemu:///system list   to check that it prompts for user + password correctly

6. Compile attached demo with "gcc -g -o saslrace saslrace.c -lvirt -lpthread"

7. ./saslrace

Actual results:
The test will often reporting that it is unable to start SASL (no mechanism available), and will also often end up crashing

eg

Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Waiting for threads to initialize
Notifying we are ready
Notifying we are ready
Telling threads to proceed
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
Segmentation fault (core dumped)


Expected results:
The test should run without errors and without crash

Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Waiting for threads to initialize
Notifying we are ready
Notifying we are ready
Telling threads to proceed
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system


Additional info:
Comment 1 Daniel Berrangé 2019-07-08 12:43:32 UTC 
Fix is upstream in

commit 7722606b369e569529ac30130f04abe17cae57cf (HEAD -> master, origin/master)
Author: Daniel P. Berrangé <berrange>
Date:   Mon Jul 8 11:32:38 2019 +0100

    rpc: ensure thread safe initialization of SASL library
Comment 6 yafu 2020-08-13 10:55:04 UTC 
Reproduced with libvirt-daemon-4.5.0-37.el8.x86_64.

Verified with libvirt-daemon-6.0.0-27.module+el8.3.0+7602+4b93512e.x86_64.
Test steps:
1. Edit /etc/libvirt/libvirtd.conf to set:
   auth_unix_rw = "sasl"

2. Edit /etc/sasl2/libvirt.conf to set:
   mech-list: scram-sha-1

3.generate sasl user and restart libvirtd service
#echo redhat |saslpasswd2 -p -a libvirt redhat
#systemctl restart libvirtd

4. virsh -c qemu:///system list   to check that it prompts for user + password correctly

5. Compile attached demo with "gcc -g -o saslrace saslrace.c -lvirt -lpthread"

6.Run './saslrace' and no coredump happened：
# ./saslrace
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Waiting for threads to initialize
Telling threads to proceed
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: test
Please enter your password: Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
...
Comment 9 errata-xmlrpc 2020-11-04 02:53:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4676
Note You need to log in before you can comment on or make changes to this bug.