Bug 1821604

Summary: [OVN SCALE] All components built/linked vs OVS source have a highly contended mutex in vlog.c
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Anton Ivanov <anivanov>
Component: openvswitchAssignee: OVN Team <ovnteam>
openvswitch sub component: other QA Contact: Jianlin Shi <jishi>
Status: MODIFIED --- Docs Contact:
Severity: unspecified    
Priority: unspecified CC: ctrautma, dcbw, qding
Version: RHEL 8.0   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openvswitch2.15-2.15.0-1.el8fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Anton Ivanov 2020-04-07 07:44:44 UTC
Any VLOG statement including debug ones in OVS will always invoke vlog_valist()

https://github.com/openvswitch/ovs/blob/master/lib/vlog.c#L1093

This function will ALWAYS lock the logging file mutex at entry regardless of the fact if it:

1. Need to log at all
2. Is logging to file

It is trivial to modify it so that it fast-paths across the "need to log" criteria and mutex-locks only if it is actually supposed to log.

This is invoked from > 2000 places including locations which are in performance critical/time critical sections.

In addition to being a performance issue, this is also an issue when trying to improve performance by splitting work between threads because it makes the codebase highly Heisenbuggy - multithreaded code works with debug on for a particular module and stops working without it.

Comment 1 Anton Ivanov 2020-04-21 09:30:56 UTC
Fix submitted upstream.

Comment 2 Anton Ivanov 2020-05-29 13:05:10 UTC
Fix is upstream, we may consider backporting it.