Bug 1727866

Summary:	Initialization of cyrus-sasl library is not thread safe causing libvirt client crashes
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	Daniel Berrangé <berrange>
Component:	libvirt	Assignee:	Daniel Berrangé <berrange>
Status:	CLOSED ERRATA	QA Contact:	yafu <yafu>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.0	CC:	chhu, dyuan, jdenemar, libvirt-maint, rbalakri, virt-bugs, xuzhang, yalzhang
Target Milestone:	rc
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-5.6.0-1.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1727865	Environment:
Last Closed:	2019-11-06 07:17:18 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1727865
Bug Blocks:

Description Daniel Berrangé 2019-07-08 11:58:52 UTC

+++ This bug was initially created as a clone of Bug #1727865 +++

Description of problem:
The sasl_client_init and/or sasl_server_init methods are required in order to initialize the SASL library global state.

Libvirt currently assumes the methods are no-ops if called more than once which is only true if the calls are serialized. The protection against multiple invocation uses a simple integer counter with no atomic access/increments, nor is any libpthread protection used. This is a clear race for threads.

Version-Release number of selected component (if applicable):
libvirt-5.0.0-11.el8

How reproducible:
Fairly often in a threaded test case

Steps to Reproduce:
1. Edit /etc/libvirt/libvirt.conf to set

   auth_unix_rw = "sasl"

2. Edit /etc/sasl2/libvirt.conf to set

   mech-list: scram-sha-1

3. As root run   "saslpasswd2 -a libvirt test"  and enter "123456" as password when prompted

4. systemctl restart libvirtd

5. virsh -c qemu:///system list   to check that it prompts for user + password correctly

6. Compile attached demo with "gcc -g -o saslrace saslrace.c -lvirt -lpthread"

7. ./saslrace

Actual results:
The test will often reporting that it is unable to start SASL (no mechanism available), and will also often end up crashing

eg

Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Waiting for threads to initialize
Notifying we are ready
Notifying we are ready
Telling threads to proceed
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Open failed
Segmentation fault (core dumped)


Expected results:
The test should run without errors and without crash

Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Waiting for threads to initialize
Notifying we are ready
Notifying we are ready
Telling threads to proceed
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system
Open qemu:///system


Additional info:

Comment 1 Daniel Berrangé 2019-07-08 12:43:51 UTC

Fix is upstream in

commit 7722606b369e569529ac30130f04abe17cae57cf (HEAD -> master, origin/master)
Author: Daniel P. Berrangé <berrange>
Date:   Mon Jul 8 11:32:38 2019 +0100

    rpc: ensure thread safe initialization of SASL library

Comment 3 yafu 2019-08-12 08:25:12 UTC

Reproduced with libvirt-5.5.0-1.el8.x86_64.

Verified with libvirt-5.6.0-1.el8.x86_64.
Test steps:
1. Edit /etc/libvirt/libvirt.conf to set:
   auth_unix_rw = "sasl"

2. Edit /etc/sasl2/libvirt.conf to set:
   mech-list: scram-sha-1

3. As root run   "saslpasswd2 -a libvirt test"  and enter "123456" as password when prompted

4. systemctl restart libvirtd

5. virsh -c qemu:///system list   to check that it prompts for user + password correctly

6. Compile attached demo with "gcc -g -o saslrace saslrace.c -lvirt -lpthread"

7.Run './saslrace' and no coredump happened：
# ./saslrace
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Notifying we are ready
Waiting for threads to initialize
Telling threads to proceed
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Opening libvirt
Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: Please enter your authentication name: test
Please enter your password: Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
Please enter your password: 
Open qemu:///system
test
Please enter your password: 
Open qemu:///system
...

Comment 5 errata-xmlrpc 2019-11-06 07:17:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723