Bug 1821604 - [OVN SCALE] All components built/linked vs OVS source have a highly contended mutex in vlog.c
Summary: [OVN SCALE] All components built/linked vs OVS source have a highly contended...
Keywords:
Status: MODIFIED
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: openvswitch
Version: RHEL 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: OVN Team
QA Contact: Jianlin Shi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-07 07:44 UTC by Anton Ivanov
Modified: 2023-07-13 07:25 UTC (History)
3 users (show)

Fixed In Version: openvswitch2.15-2.15.0-1.el8fdp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FD-558 0 None None None 2022-02-16 16:00:59 UTC

Description Anton Ivanov 2020-04-07 07:44:44 UTC
Any VLOG statement including debug ones in OVS will always invoke vlog_valist()

https://github.com/openvswitch/ovs/blob/master/lib/vlog.c#L1093

This function will ALWAYS lock the logging file mutex at entry regardless of the fact if it:

1. Need to log at all
2. Is logging to file

It is trivial to modify it so that it fast-paths across the "need to log" criteria and mutex-locks only if it is actually supposed to log.

This is invoked from > 2000 places including locations which are in performance critical/time critical sections.

In addition to being a performance issue, this is also an issue when trying to improve performance by splitting work between threads because it makes the codebase highly Heisenbuggy - multithreaded code works with debug on for a particular module and stops working without it.

Comment 1 Anton Ivanov 2020-04-21 09:30:56 UTC
Fix submitted upstream.

Comment 2 Anton Ivanov 2020-05-29 13:05:10 UTC
Fix is upstream, we may consider backporting it.


Note You need to log in before you can comment on or make changes to this bug.