This service will be undergoing maintenance at 03:30 UTC, 2016-05-27. It is expected to last about 2 hours
Bug 983026 - Only open netcf once in libvirtd
Only open netcf once in libvirtd
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Laine Stump
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-10 06:16 EDT by Daniel Berrange
Modified: 2014-06-17 20:52 EDT (History)
8 users (show)

See Also:
Fixed In Version: libvirt-1.1.1-5.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-13 07:36:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Daniel Berrange 2013-07-10 06:16:04 EDT
Description of problem:
Currently each client connecting to libvirtd will cause a new netcf handle to be opened. This is inefficient in terms of memory usage, and more critically each netcf handle consumes a netlink socket handle. There are only ~1000 netlink handles available per process, so this usage of netlink sockets places a limit on the number of concurrent connections libvirtd can handle. We must change to open netcf once per libvirtd. The slight complication here is that we'll need a way to refresh the NIC configs periodically, so we can cope with changes behind libvirt's back. Arguably this problem already exists if an application keeps a libvirt connection open for a long term.

Version-Release number of selected component (if applicable):
1.1.0-1.el7

How reproducible:
Always

Steps to Reproduce:
1. Open 20 connections to libvirtd
2. Look at number of open netlink sockets (lsof)
3.

Actual results:
One netlink socket per connection

Expected results:
One netlink socket per libvirtd

Additional info:
Comment 2 Laine Stump 2013-08-04 21:22:34 EDT
(In reply to Daniel Berrange from comment #0)
> We must change to open netcf once per libvirtd.

Just adding a simple state driver that saves the fd for the netcf socket, then makes sure that all access to that socket is protected by a mutex should be enough, right? Let me know if I'm missing something obvious...

> The slight
> complication here is that we'll need a way to refresh the NIC configs
> periodically, so we can cope with changes behind libvirt's back.

The virInterface calls don't cache any information from netcf and if I recall correctly, the cache that netcf keeps is sent back to augeas each time something is requested from netcf (see the function get_augeas() in the netcf source) and augeas checks the modification dates of each file and only rereads those files that have been touched since the data was last gathered. So I don't think this will be (or is) a problem.

You'll notice this updating in action if you run virt-manager and look at the list of interfaces while adding/removing interfaces - they show up almost immediately.
Comment 3 Daniel Berrange 2013-08-05 05:53:09 EDT
(In reply to Laine Stump from comment #2)
> (In reply to Daniel Berrange from comment #0)
> > We must change to open netcf once per libvirtd.
> 
> Just adding a simple state driver that saves the fd for the netcf socket,
> then makes sure that all access to that socket is protected by a mutex
> should be enough, right? Let me know if I'm missing something obvious...

Well if by 'fd', you mean the 'struct netcf' instance, then yes. The APIs already have locking present, since even a single virConnectPtr needs thread protection.

> > The slight
> > complication here is that we'll need a way to refresh the NIC configs
> > periodically, so we can cope with changes behind libvirt's back.
> 
> The virInterface calls don't cache any information from netcf and if I
> recall correctly, the cache that netcf keeps is sent back to augeas each
> time something is requested from netcf (see the function get_augeas() in the
> netcf source) and augeas checks the modification dates of each file and only
> rereads those files that have been touched since the data was last gathered.
> So I don't think this will be (or is) a problem.
> 
> You'll notice this updating in action if you run virt-manager and look at
> the list of interfaces while adding/removing interfaces - they show up
> almost immediately.

Ah, I thought that it only parsed the ifcfg-XXX files when you did 'netcf_init', but if that's wrong, then this bug is easy.
Comment 4 Laine Stump 2013-08-05 06:27:27 EDT
(In reply to Daniel Berrange from comment #3)
> (In reply to Laine Stump from comment #2)
> > (In reply to Daniel Berrange from comment #0)
> > > We must change to open netcf once per libvirtd.
> > 
> > Just adding a simple state driver that saves the fd for the netcf socket,
> > then makes sure that all access to that socket is protected by a mutex
> > should be enough, right? Let me know if I'm missing something obvious...
> 
> Well if by 'fd', you mean the 'struct netcf' instance, then yes. The APIs
> already have locking present, since even a single virConnectPtr needs thread
> protection.

Yeah, that's what I meant. Not enough sleep lately :-)

Good, I thought that was all it would take but wanted to make sure before promising anything. I agree that this will be easy to fix, so I'll get to it as soon as possible (not until next week, though).
Comment 5 Laine Stump 2013-09-12 08:35:43 EDT
The following two patches have been pushed upstream to resolve this bug:

commit 4c5fa430972246b82d60c875654fc57e48adfcce
Author: Laine Stump <laine@laine.org>
Date:   Tue Aug 27 12:26:48 2013 -0400

    rename "struct interface_driver" to virNetcfDriverState

commit 822fe1367dd55e3de7c3c044ee02fd733ab3415a
Author: Laine Stump <laine@laine.org>
Date:   Wed Aug 28 12:56:21 2013 -0400

    netcf driver: use a single netcf handle for all connections
Comment 7 Jincheng Miao 2013-09-17 23:42:12 EDT
The latest libvirt will only open one netlink for netcf, other than previous libvirt will open multiple netlinks, just like:

1. prepare a test script
# cat test.py 
import time
import libvirt
con=[]
for i in range(1,20):
	con.append(libvirt.open(None))
time.sleep(1000)

2. for previous libvirt
# rpm -q libvirt
libvirt-1.1.1-4.el7.x86_64

# lsof -p `pidof libvirtd` | grep netlink
libvirtd 5136 root   13u  netlink                          0t0      45202 KOBJECT_UEVENT
libvirtd 5136 root   14u  netlink                          0t0      45201 ROUTE
libvirtd 5136 root   17u  netlink                          0t0      38689 KOBJECT_UEVENT

# python test.py &

# lsof -p `pidof libvirtd` | grep netlink
libvirtd 5136 root   13u  netlink                          0t0      45202 KOBJECT_UEVENT
libvirtd 5136 root   14u  netlink                          0t0      45201 ROUTE
libvirtd 5136 root   17u  netlink                          0t0      38689 KOBJECT_UEVENT
libvirtd 5136 root   21u  netlink                          0t0      39627 ROUTE
libvirtd 5136 root   24u  netlink                          0t0      39632 ROUTE
libvirtd 5136 root   27u  netlink                          0t0      43714 ROUTE
libvirtd 5136 root   30u  netlink                          0t0      38702 ROUTE
libvirtd 5136 root   33u  netlink                          0t0      44446 ROUTE
libvirtd 5136 root   36u  netlink                          0t0      39656 ROUTE
libvirtd 5136 root   39u  netlink                          0t0      43724 ROUTE
libvirtd 5136 root   42u  netlink                          0t0      38717 ROUTE
libvirtd 5136 root   45u  netlink                          0t0      38721 ROUTE
...

3. for latest libvirt
# rpm -q libvirt
libvirt-1.1.1-5.el7.x86_64

# lsof -p `pidof libvirtd` | grep netlink
libvirtd 5402 root   13u  netlink                          0t0      44751 ROUTE
libvirtd 5402 root   14u  netlink                          0t0      44752 KOBJECT_UEVENT
libvirtd 5402 root   17u  netlink                          0t0      46348 KOBJECT_UEVENT
libvirtd 5402 root   20u  netlink                          0t0      38790 ROUTE

# python test.py &

# lsof -p `pidof libvirtd` | grep netlink
libvirtd 5402 root   13u  netlink                          0t0      44751 ROUTE
libvirtd 5402 root   14u  netlink                          0t0      44752 KOBJECT_UEVENT
libvirtd 5402 root   17u  netlink                          0t0      46348 KOBJECT_UEVENT
libvirtd 5402 root   20u  netlink                          0t0      38790 ROUTE


This fix can handle netlink multi-opened problem, so I change the status to VERIFIED.
Comment 8 Ludek Smid 2014-06-13 07:36:36 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.