Bug 861069
| Summary: | Potential libvirtd segfault in netcf driver | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Vladislav Bogdanov <bubble> | ||||
| Component: | libvirt | Assignee: | Libvirt Maintainers <libvirt-maint> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 17 | CC: | berrange, clalancette, crobinso, itamar, jforbes, jyang, laine, libvirt-maint, veillard, virt-maint | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2012-11-01 14:48:49 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Vladislav Bogdanov
2012-09-27 13:08:40 UTC
Created attachment 618048 [details]
Rough patch to fix mutex issue
Attached patch fixes issue by moving mutex initialization from interfaceOpenInterface() to interfaceRegister().
Note: patch may fuzz because it is hand-edited from previous libvirt versions, but applies fine to 0.10.2
No global driver lock is needed, because netcf itself is re-entrant (as long as each thread separately calls ncf_init, and uses the netcf object that it returns for all subsequent calls). If you are experiencing a problem that is due to multiple libvirt threads being in netcf at once, then the problem is much more likely due to: 1) a re-entrancy bug in netcf or one of its libraries (we have previously found thread-unsafe code in both augeas and libnl as a result of calling them from netcf), or in netcf itself. 2) a bug in the per-connection locking that is in libvirt. Have you experienced a crash that you can attribute to multi-threaded use of virInterface functions? Can you describe it in more detail? Even if you're unable to get a backtrace, perhaps a description of how you trigger it could aid someone else in debugging. (In reply to comment #2) > No global driver lock is needed, because netcf itself is re-entrant (as long > as each thread separately calls ncf_init, and uses the netcf object that it > returns for all subsequent calls). 1. Is that true for ncf_init() itself? 2. What then that per-thread mutex protects from? > If you are experiencing a problem that is due to multiple libvirt threads > being in netcf at once, then the problem is much more likely due to: > > 1) a re-entrancy bug in netcf or one of its libraries (we have previously > found thread-unsafe code in both augeas and libnl as a result of calling > them from netcf), or in netcf itself. > > 2) a bug in the per-connection locking that is in libvirt. Perfectly correct. I bet 1) is true, see below. > Have you experienced a crash that you can attribute to multi-threaded use of > virInterface functions? Can you describe it in more detail? Yes, as long as system has plenty of interfaces (I use bridges on top of vlans on top of LACP bond on top of two intel cards) and many connections are handled constantly. > Even if you're > unable to get a backtrace, perhaps a description of how you trigger it could > aid someone else in debugging. System has more than dozen of bridges which are listed as intrfaces. System runs pacemaker which has resources for several disk pools (4 in my case), some are LVM based (clvm), some are filesystem-based (CIFS). Also, several dozens of virtual machines is run over the cluster, so every node runs 10-20 VMs. Pacemaker monitors each libvirt-related resource several times per minute (2-6). Each monitor operation results in connection being made to local libvirtd daemon, so I'd say libvirtd load in terms of connection handling (connect, do the request, disconnect) is moderate. I should admit that no calls are made directly to 'interface' driver, but driver is initialized once per connection. If I recall that correctly, SIGSEGVs I saw happened somewhere inside of ncf_init(). So, may be only that function should be serialized, not all calls. I saw that crashes several times per day after installation of first libvirt release with interface driver (iirc it was added after 0.8.1). After I moved mutex to global space and serialized calls to netcf, libvirtd does not crash at all anymore (for >6 months). (In reply to comment #3) > I saw that crashes several times per day after installation of first libvirt > release with interface driver (iirc it was added after 0.8.1). s/with interface driver/compiled with netcf support/ > I saw that crashes several times per day after installation of first libvirt
> release with interface driver (iirc it was added after 0.8.1).
> After I moved mutex to global space and serialized calls to netcf, libvirtd
> does not crash at all anymore (for >6 months).
I'm willing to be that those crashes were due to the libnl thread safety bugs we discovered.
Are that bugs fixed? I see only 1.1 in koji. And nothing relevant in a changelog. Vladislav, in F18 at least libvirt uses libnl3, which is a different package. Can you drop your local patch, reproduce one of the crashes, and post the stack trace here so we can confirm if this is a known issue? Okay, i'm just assuming modern libnl fixes this. Closing as CURRENTRELEASE |