Bug 1727866 - Initialization of cyrus-sasl library is not thread safe causing libvirt client crashes
Summary: Initialization of cyrus-sasl library is not thread safe causing libvirt clien...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Daniel Berrangé
QA Contact: yafu
URL:
Whiteboard:
Depends On: 1727865
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-08 11:58 UTC by Daniel Berrangé
Modified: 2020-11-14 05:35 UTC (History)
8 users (show)

Fixed In Version: libvirt-5.6.0-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1727865
Environment:
Last Closed: 2019-11-06 07:17:18 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3723 0 None None None 2019-11-06 07:17:38 UTC

Description Daniel Berrangé 2019-07-08 11:58:52 UTC
+++ This bug was initially created as a clone of Bug #1727865 +++

Description of problem:
The sasl_client_init and/or sasl_server_init methods are required in order to initialize the SASL library global state.

Libvirt currently assumes the methods are no-ops if called more than once which is only true if the calls are serialized. The protection against multiple invocation uses a simple integer counter with no atomic access/increments, nor is any libpthread protection used. This is a clear race for threads.

Version-Release number of selected component (if applicable):
libvirt-5.0.0-11.el8

How reproducible:
Fairly often in a threaded test case

Steps to Reproduce:
1. Edit /etc/libvirt/libvirt.conf to set

   auth_unix_rw = "sasl"

2. Edit /etc/sasl2/libvirt.conf to set

   mech-list: scram-sha-1

3. As root run   "saslpasswd2 -a libvirt test"  and enter "123456" as password when prompted

4. systemctl restart libvirtd

5. virsh -c qemu:///system list   to check that it prompts for user + password correctly

6. Compile attached demo with "gcc -g -o saslrace saslrace.c -lvirt -lpthread"

7. ./saslrace

Actual results:
The test will often reporting that it is unable to start SASL (no mechanism available), and will also often end up crashing

eg

Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Waiting for threads to initialize
Notifying we are ready
Notifying we are ready
Telling threads to proceed
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
Segmentation fault (core dumped)


Expected results:
The test should run without errors and without crash

Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Waiting for threads to initialize
Notifying we are ready
Notifying we are ready
Telling threads to proceed
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system


Additional info:

Comment 1 Daniel Berrangé 2019-07-08 12:43:51 UTC
Fix is upstream in

commit 7722606b369e569529ac30130f04abe17cae57cf (HEAD -> master, origin/master)
Author: Daniel P. Berrangé <berrange>
Date:   Mon Jul 8 11:32:38 2019 +0100

    rpc: ensure thread safe initialization of SASL library

Comment 3 yafu 2019-08-12 08:25:12 UTC
Reproduced with libvirt-5.5.0-1.el8.x86_64.

Verified with libvirt-5.6.0-1.el8.x86_64.
Test steps:
1. Edit /etc/libvirt/libvirt.conf to set:
   auth_unix_rw = "sasl"

2. Edit /etc/sasl2/libvirt.conf to set:
   mech-list: scram-sha-1

3. As root run   "saslpasswd2 -a libvirt test"  and enter "123456" as password when prompted

4. systemctl restart libvirtd

5. virsh -c qemu:///system list   to check that it prompts for user + password correctly

6. Compile attached demo with "gcc -g -o saslrace saslrace.c -lvirt -lpthread"

7.Run './saslrace' and no coredump happened:
# ./saslrace
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Waiting for threads to initialize
Telling threads to proceed
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: test
Please enter your password: Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
...

Comment 5 errata-xmlrpc 2019-11-06 07:17:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723


Note You need to log in before you can comment on or make changes to this bug.