RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 983026 - Only open netcf once in libvirtd
Summary: Only open netcf once in libvirtd
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-10 10:16 UTC by Daniel Berrangé
Modified: 2014-06-18 00:52 UTC (History)
8 users (show)

Fixed In Version: libvirt-1.1.1-5.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-13 11:36:36 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Daniel Berrangé 2013-07-10 10:16:04 UTC
Description of problem:
Currently each client connecting to libvirtd will cause a new netcf handle to be opened. This is inefficient in terms of memory usage, and more critically each netcf handle consumes a netlink socket handle. There are only ~1000 netlink handles available per process, so this usage of netlink sockets places a limit on the number of concurrent connections libvirtd can handle. We must change to open netcf once per libvirtd. The slight complication here is that we'll need a way to refresh the NIC configs periodically, so we can cope with changes behind libvirt's back. Arguably this problem already exists if an application keeps a libvirt connection open for a long term.

Version-Release number of selected component (if applicable):
1.1.0-1.el7

How reproducible:
Always

Steps to Reproduce:
1. Open 20 connections to libvirtd
2. Look at number of open netlink sockets (lsof)
3.

Actual results:
One netlink socket per connection

Expected results:
One netlink socket per libvirtd

Additional info:

Comment 2 Laine Stump 2013-08-05 01:22:34 UTC
(In reply to Daniel Berrange from comment #0)
> We must change to open netcf once per libvirtd.

Just adding a simple state driver that saves the fd for the netcf socket, then makes sure that all access to that socket is protected by a mutex should be enough, right? Let me know if I'm missing something obvious...

> The slight
> complication here is that we'll need a way to refresh the NIC configs
> periodically, so we can cope with changes behind libvirt's back.

The virInterface calls don't cache any information from netcf and if I recall correctly, the cache that netcf keeps is sent back to augeas each time something is requested from netcf (see the function get_augeas() in the netcf source) and augeas checks the modification dates of each file and only rereads those files that have been touched since the data was last gathered. So I don't think this will be (or is) a problem.

You'll notice this updating in action if you run virt-manager and look at the list of interfaces while adding/removing interfaces - they show up almost immediately.

Comment 3 Daniel Berrangé 2013-08-05 09:53:09 UTC
(In reply to Laine Stump from comment #2)
> (In reply to Daniel Berrange from comment #0)
> > We must change to open netcf once per libvirtd.
> 
> Just adding a simple state driver that saves the fd for the netcf socket,
> then makes sure that all access to that socket is protected by a mutex
> should be enough, right? Let me know if I'm missing something obvious...

Well if by 'fd', you mean the 'struct netcf' instance, then yes. The APIs already have locking present, since even a single virConnectPtr needs thread protection.

> > The slight
> > complication here is that we'll need a way to refresh the NIC configs
> > periodically, so we can cope with changes behind libvirt's back.
> 
> The virInterface calls don't cache any information from netcf and if I
> recall correctly, the cache that netcf keeps is sent back to augeas each
> time something is requested from netcf (see the function get_augeas() in the
> netcf source) and augeas checks the modification dates of each file and only
> rereads those files that have been touched since the data was last gathered.
> So I don't think this will be (or is) a problem.
> 
> You'll notice this updating in action if you run virt-manager and look at
> the list of interfaces while adding/removing interfaces - they show up
> almost immediately.

Ah, I thought that it only parsed the ifcfg-XXX files when you did 'netcf_init', but if that's wrong, then this bug is easy.

Comment 4 Laine Stump 2013-08-05 10:27:27 UTC
(In reply to Daniel Berrange from comment #3)
> (In reply to Laine Stump from comment #2)
> > (In reply to Daniel Berrange from comment #0)
> > > We must change to open netcf once per libvirtd.
> > 
> > Just adding a simple state driver that saves the fd for the netcf socket,
> > then makes sure that all access to that socket is protected by a mutex
> > should be enough, right? Let me know if I'm missing something obvious...
> 
> Well if by 'fd', you mean the 'struct netcf' instance, then yes. The APIs
> already have locking present, since even a single virConnectPtr needs thread
> protection.

Yeah, that's what I meant. Not enough sleep lately :-)

Good, I thought that was all it would take but wanted to make sure before promising anything. I agree that this will be easy to fix, so I'll get to it as soon as possible (not until next week, though).

Comment 5 Laine Stump 2013-09-12 12:35:43 UTC
The following two patches have been pushed upstream to resolve this bug:

commit 4c5fa430972246b82d60c875654fc57e48adfcce
Author: Laine Stump <laine>
Date:   Tue Aug 27 12:26:48 2013 -0400

    rename "struct interface_driver" to virNetcfDriverState

commit 822fe1367dd55e3de7c3c044ee02fd733ab3415a
Author: Laine Stump <laine>
Date:   Wed Aug 28 12:56:21 2013 -0400

    netcf driver: use a single netcf handle for all connections

Comment 7 Jincheng Miao 2013-09-18 03:42:12 UTC
The latest libvirt will only open one netlink for netcf, other than previous libvirt will open multiple netlinks, just like:

1. prepare a test script
# cat test.py 
import time
import libvirt
con=[]
for i in range(1,20):
	con.append(libvirt.open(None))
time.sleep(1000)

2. for previous libvirt
# rpm -q libvirt
libvirt-1.1.1-4.el7.x86_64

# lsof -p `pidof libvirtd` | grep netlink
libvirtd 5136 root   13u  netlink                          0t0      45202 KOBJECT_UEVENT
libvirtd 5136 root   14u  netlink                          0t0      45201 ROUTE
libvirtd 5136 root   17u  netlink                          0t0      38689 KOBJECT_UEVENT

# python test.py &

# lsof -p `pidof libvirtd` | grep netlink
libvirtd 5136 root   13u  netlink                          0t0      45202 KOBJECT_UEVENT
libvirtd 5136 root   14u  netlink                          0t0      45201 ROUTE
libvirtd 5136 root   17u  netlink                          0t0      38689 KOBJECT_UEVENT
libvirtd 5136 root   21u  netlink                          0t0      39627 ROUTE
libvirtd 5136 root   24u  netlink                          0t0      39632 ROUTE
libvirtd 5136 root   27u  netlink                          0t0      43714 ROUTE
libvirtd 5136 root   30u  netlink                          0t0      38702 ROUTE
libvirtd 5136 root   33u  netlink                          0t0      44446 ROUTE
libvirtd 5136 root   36u  netlink                          0t0      39656 ROUTE
libvirtd 5136 root   39u  netlink                          0t0      43724 ROUTE
libvirtd 5136 root   42u  netlink                          0t0      38717 ROUTE
libvirtd 5136 root   45u  netlink                          0t0      38721 ROUTE
...

3. for latest libvirt
# rpm -q libvirt
libvirt-1.1.1-5.el7.x86_64

# lsof -p `pidof libvirtd` | grep netlink
libvirtd 5402 root   13u  netlink                          0t0      44751 ROUTE
libvirtd 5402 root   14u  netlink                          0t0      44752 KOBJECT_UEVENT
libvirtd 5402 root   17u  netlink                          0t0      46348 KOBJECT_UEVENT
libvirtd 5402 root   20u  netlink                          0t0      38790 ROUTE

# python test.py &

# lsof -p `pidof libvirtd` | grep netlink
libvirtd 5402 root   13u  netlink                          0t0      44751 ROUTE
libvirtd 5402 root   14u  netlink                          0t0      44752 KOBJECT_UEVENT
libvirtd 5402 root   17u  netlink                          0t0      46348 KOBJECT_UEVENT
libvirtd 5402 root   20u  netlink                          0t0      38790 ROUTE


This fix can handle netlink multi-opened problem, so I change the status to VERIFIED.

Comment 8 Ludek Smid 2014-06-13 11:36:36 UTC
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.


Note You need to log in before you can comment on or make changes to this bug.