Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 992980

Summary: Separate limits for anonymous and authenticated users
Product: Red Hat Enterprise Linux 7 Reporter: Michal Privoznik <mprivozn>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0CC: ajia, berrange, dyuan, gsun, lsu, rbalakri, rjones, weizhan, yanyang, zpeng
Target Milestone: rcKeywords: Upstream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.2.7-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 981729
: 1086175 (view as bug list) Environment:
Last Closed: 2015-03-05 07:24:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 981729, 1014604, 1058606, 1070221    
Bug Blocks: 1086175    

Description Michal Privoznik 2013-08-05 10:48:45 UTC 
+++ This bug was initially created as a clone of Bug #981729 +++

Description of problem:
The 'max_clients' setting controls how many clients are allowed to connect to libvirt, as a protection against DOS attacker from unauthenticated users.

It is problematic when dealing with concurrent start of large numbers of containers, because this can trigger a a very large number of connections in a short period of time.

With the default max_clients=20, this easily causes container startup failure.

We can't simply raise the limit since it is DOS protection, but we can improve the behaviour somewhat.

Currently libvirt will unconditionally accept() any incoming socket, regardless of the current number of clients. Thus if the max limit is hit, libvirt will accept and immediately  close client connections.

It would be better if libvirt simply did not accept() the connection. This would let pending connections wait for a previous connection to close before continuing. The max number of queued connections can be controlled via the listen() syscall, and could be a fairly large number, since they consume minimal resources. This would be a new "max_queued_clients" setting in libvirtd.conf, perhaps as much as 1000 

A new 'max_anonymous_clients' setting could limit only those connections which are accept()ed, but not yet authenticated. This could be fairly low (perhaps current 20)

The existing 'max_clients' setting could then be used as a limit on total number of connections, and set to a far higher value (perhaps several 100 or more)

Version-Release number of selected component (if applicable):
1.1.0-1.el6.

How reproducible:
Always

Steps to Reproduce:
1. Attempt to open 30 concurrent connections to libvirt
2.
3.

Actual results:
Only first 20 succeed, the rest are dropped

Expected results:
10 connections are queued, pending close of 10 earlier connections

Additional info:

--- Additional comment from RHEL Product and Program Management on 2013-07-05 17:47:50 CEST ---

Since this bug report was entered in bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Alex Jia on 2013-07-08 12:23:37 CEST ---

# tail -2 /etc/libvirt/libvirtd.conf
max_clients = 20
max_workers = 20


# for i in {1..30}; do virt-sandbox-service create -C -u httpd.service -N dhcp myapache$i;done

# for i in {1..30}; do virt-sandbox-service create start myapache$i & done

XXX

# Unable to open connection: Unable to open lxc:///: Cannot recv data: Connection reset by peer
Unable to open connection: Unable to open lxc:///: Cannot recv data: Connection reset by peer
Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe
Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe
Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe
Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe
Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe
Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe
Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe
Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe

And check libvirtd log:

2013-07-08 10:13:58.933+0000: 8034: error : virNetServerAddClient:262 : Too many active clients (20), dropping connection from 127.0.0.1;0
2013-07-08 10:13:58.941+0000: 8034: error : virNetServerAddClient:262 : Too many active clients (20), dropping connection from 127.0.0.1;0
2013-07-08 10:13:58.943+0000: 8034: error : virNetServerAddClient:262 : Too many active clients (20), dropping connection from 127.0.0.1;0

--- Additional comment from Michal Privoznik on 2013-07-25 16:24:16 CEST ---

I've just proposed patches upstream:

https://www.redhat.com/archives/libvir-list/2013-July/msg01646.html

--- Additional comment from Michal Privoznik on 2013-08-05 11:31:02 CEST ---

Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2013-August/msg00079.html

--- Additional comment from Daniel Berrange on 2013-08-05 11:58:33 CEST ---

NB the upstream patches only implement half of this bug. There's still no separation of the limits for anonymous vs authenticated clients.


This bug is to address the second part of original request = the 'max_anonymous_clients' setting.
Comment 2 Michal Privoznik 2013-12-09 14:36:57 UTC 
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2013-December/msg00453.html
Comment 3 Michal Privoznik 2014-03-17 16:51:33 UTC 
I've just pushed patches upstream:

commit 68f60f669c566e53904ce39c95a57853f7f23638
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Mar 4 18:55:24 2014 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Mon Mar 17 17:45:13 2014 +0100

    daemon: Introduce max_anonymous_clients
    
    https://bugzilla.redhat.com/show_bug.cgi?id=992980
    
    This config tunable allows users to determine the maximum number of
    accepted but yet not authenticated users.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 4015396b2cf9f43ff77d24e3a4d3e1372f5352a3
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Mar 4 15:37:27 2014 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Mon Mar 17 17:37:42 2014 +0100

    virNetServer: Introduce unauth clients counter
    
    The counter gets incremented on each unauthenticated client added to the
    server and decremented whenever the client authenticates.
    
    Signed-off-by: Michal Privoznik <mprivozn>


v1.2.2-191-g68f60f6
Comment 5 Yang Yang 2014-10-30 09:26:16 UTC 
Hi Michal,

I set 'max_anonymous_clients' to '2' in libvirtd.conf. Then I tried to start 20 LXC containers. All of them are started. There is no error about 'dropping connection' in libvirt log. Could you please help check whether It can be verified by the following steps or not?

Thanks in advance.
Yang

# rpm -q libvirt
libvirt-1.2.8-5.el7.x86_64

# grep max /etc/libvirt/libvirtd.conf
max_clients= 4 
#max_queued_clients = 2 
max_anonymous_clients = 2 
max_workers =4 


# virsh -c lxc:/// list --all
 Id    Name                           State
----------------------------------------------------
 -     lxc-test-0                     shut off
 -     lxc-test-1                     shut off
 -     lxc-test-10                    shut off
 -     lxc-test-11                    shut off
 -     lxc-test-12                    shut off
 -     lxc-test-13                    shut off
 -     lxc-test-14                    shut off
 -     lxc-test-15                    shut off
 -     lxc-test-16                    shut off
 -     lxc-test-17                    shut off
 -     lxc-test-18                    shut off
 -     lxc-test-19                    shut off
 -     lxc-test-2                     shut off
 -     lxc-test-20                    shut off
 -     lxc-test-3                     shut off
 -     lxc-test-4                     shut off
 -     lxc-test-5                     shut off
 -     lxc-test-6                     shut off
 -     lxc-test-7                     shut off
 -     lxc-test-8                     shut off
 -     lxc-test-9                     shut off

# for i in {1..20}; do virsh -c lxc:/// start lxc-test-$i & done

# virsh -c lxc:/// list 
 Id    Name                           State
----------------------------------------------------
 22004 lxc-test-13                    running
 22025 lxc-test-1                     running
 22031 lxc-test-7                     running
 22037 lxc-test-12                    running
 22049 lxc-test-5                     running
 22069 lxc-test-3                     running
 22078 lxc-test-18                    running
 22082 lxc-test-20                    running
 22102 lxc-test-16                    running
 22110 lxc-test-4                     running
 22113 lxc-test-11                    running
 22134 lxc-test-14                    running
 22142 lxc-test-6                     running
 22145 lxc-test-2                     running
 22156 lxc-test-17                    running
 22179 lxc-test-8                     running
 22186 lxc-test-9                     running
 22191 lxc-test-19                    running
 22202 lxc-test-15                    running
 22217 lxc-test-10                    running

[root@rhel7_test yy]# grep virNetServerAddClient /var/log/libvirtd.log 
2014-10-30 08:40:49.111+0000: 20961: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients
2014-10-30 08:40:49.125+0000: 20961: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients
2014-10-30 08:40:49.127+0000: 20961: debug : virNetServerAddClient:295 : Temporarily suspending services due to max_clients
2014-10-30 08:41:19.346+0000: 20961: debug : virNetServerAddClient:295 : Temporarily suspending services due to max_clients
2014-10-30 08:41:49.574+0000: 20961: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients
2014-10-30 08:41:49.574+0000: 20961: debug : virNetServerAddClient:295 : Temporarily suspending services due to max_clients
...snip...

[root@rhel7_test yy]# grep virNetServerCheckLimits /var/log/libvirtd.log 
2014-10-30 08:40:49.113+0000: 20965: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=2 nclients_max=4 nclients_unauth=1 nclients_unauth_max=2
2014-10-30 08:40:49.113+0000: 20965: debug : virNetServerCheckLimits:1083 : Re-enabling services
2014-10-30 08:40:49.126+0000: 20970: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=3 nclients_max=4 nclients_unauth=1 nclients_unauth_max=2
2014-10-30 08:40:49.126+0000: 20970: debug : virNetServerCheckLimits:1083 : Re-enabling services
2014-10-30 08:40:49.127+0000: 20970: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=3 nclients_max=4 nclients_unauth=0 nclients_unauth_max=2
2014-10-30 08:40:49.127+0000: 20970: debug : virNetServerCheckLimits:1083 : Re-enabling services
2014-10-30 08:40:49.128+0000: 20968: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=4 nclients_max=4 nclients_unauth=0 nclients_unauth_max=2
2014-10-30 08:41:19.343+0000: 20961: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=3 nclients_max=4 nclients_unauth=0 nclients_unauth_max=2
2014-10-30 08:41:19.343+0000: 20961: debug : virNetServerCheckLimits:1083 : Re-enabling services
...snip...
Comment 6 Michal Privoznik 2014-10-30 13:14:25 UTC 
(In reply to yangyang from comment #5)
> Hi Michal,
> 
> I set 'max_anonymous_clients' to '2' in libvirtd.conf. Then I tried to start
> 20 LXC containers. All of them are started. There is no error about
> 'dropping connection' in libvirt log. Could you please help check whether It
> can be verified by the following steps or not?
> 
> Thanks in advance.
> Yang
> 
> # rpm -q libvirt
> libvirt-1.2.8-5.el7.x86_64
> 
> # grep max /etc/libvirt/libvirtd.conf
> max_clients= 4 
> #max_queued_clients = 2 
> max_anonymous_clients = 2 
> max_workers =4 

This could be used, however, you need to set max_queued_clients=0. The problem is, even though libvirt doesn't accept incoming client on the socket, kernel will do parial opening, and queue clients on the socket from which they are taken off by calling accept(). The size of the queue is managed by max_queued_clients. So to disable this set it to zero and you should start seeing connection errors.
Comment 7 Yang Yang 2014-10-31 06:03:39 UTC 
(In reply to Michal Privoznik from comment #6)
> (In reply to yangyang from comment #5)
> > Hi Michal,
> > 
> > I set 'max_anonymous_clients' to '2' in libvirtd.conf. Then I tried to start
> > 20 LXC containers. All of them are started. There is no error about
> > 'dropping connection' in libvirt log. Could you please help check whether It
> > can be verified by the following steps or not?
> > 
> > Thanks in advance.
> > Yang
> > 
> > # rpm -q libvirt
> > libvirt-1.2.8-5.el7.x86_64
> > 
> > # grep max /etc/libvirt/libvirtd.conf
> > max_clients= 4 
> > #max_queued_clients = 2 
> > max_anonymous_clients = 2 
> > max_workers =4 
> 
> This could be used, however, you need to set max_queued_clients=0. The
> problem is, even though libvirt doesn't accept incoming client on the
> socket, kernel will do parial opening, and queue clients on the socket from
> which they are taken off by calling accept(). The size of the queue is
> managed by max_queued_clients. So to disable this set it to zero and you
> should start seeing connection errors.

If 'max_queued_clients' is set to zero, it will be translated into 30, right?

If 'max_anonymous_clients' is set to '-1', it will be translated into '18446744073709551615', is it expected result ?

for example:
# grep max /etc/libvirt/libvirtd.conf
max_clients= 40
max_queued_clients = 0
max_anonymous_clients = -1
max_workers =40

#grep virNetServerCheckLimits /var/log/libvirtd.log
2014-10-31 05:38:14.146+0000: 29136: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=0 nclients_max=40 nclients_unauth=0 nclients_unauth_max=18446744073709551615
2014-10-31 05:38:14.146+0000: 29136: debug : virNetServerCheckLimits:1083 : Re-enabling services

Thanks 
Yang
Comment 8 Michal Privoznik 2014-10-31 13:41:25 UTC 
(In reply to yangyang from comment #7)
> (In reply to Michal Privoznik from comment #6)
> > (In reply to yangyang from comment #5)
> > > Hi Michal,
> > > 
> > > I set 'max_anonymous_clients' to '2' in libvirtd.conf. Then I tried to start
> > > 20 LXC containers. All of them are started. There is no error about
> > > 'dropping connection' in libvirt log. Could you please help check whether It
> > > can be verified by the following steps or not?
> > > 
> > > Thanks in advance.
> > > Yang
> > > 
> > > # rpm -q libvirt
> > > libvirt-1.2.8-5.el7.x86_64
> > > 
> > > # grep max /etc/libvirt/libvirtd.conf
> > > max_clients= 4 
> > > #max_queued_clients = 2 
> > > max_anonymous_clients = 2 
> > > max_workers =4 
> > 
> > This could be used, however, you need to set max_queued_clients=0. The
> > problem is, even though libvirt doesn't accept incoming client on the
> > socket, kernel will do parial opening, and queue clients on the socket from
> > which they are taken off by calling accept(). The size of the queue is
> > managed by max_queued_clients. So to disable this set it to zero and you
> > should start seeing connection errors.
> 
> If 'max_queued_clients' is set to zero, it will be translated into 30, right?

Correct.

> 
> If 'max_anonymous_clients' is set to '-1', it will be translated into
> '18446744073709551615', is it expected result ?

Well, it's a broader problem I think. I mean, we don't check for negative values in other cases too (e.g. all these max_*). So I'd save it for separate bug. It shouldn't be a show stopper for this feature.

> 
> for example:
> # grep max /etc/libvirt/libvirtd.conf
> max_clients= 40
> max_queued_clients = 0
> max_anonymous_clients = -1
> max_workers =40
> 
> #grep virNetServerCheckLimits /var/log/libvirtd.log
> 2014-10-31 05:38:14.146+0000: 29136: debug : virNetServerCheckLimits:1078 :
> Considering re-enabling services: nclients=0 nclients_max=40
> nclients_unauth=0 nclients_unauth_max=18446744073709551615
> 2014-10-31 05:38:14.146+0000: 29136: debug : virNetServerCheckLimits:1083 :
> Re-enabling services
> 

So it works, nice.
Comment 9 Yang Yang 2014-11-06 05:57:44 UTC 
Opened a separate bug to track the negative value issue
https://bugzilla.redhat.com/show_bug.cgi?id=1160995

Verify this one as following

Product version
libvirt-1.2.8-6.el7.x86_64

Steps
1. set 'max_*' as following
# grep max /etc/libvirt/libvirtd.conf
max_clients= 40
max_queued_clients = 0
max_anonymous_clients = 2
max_workers = 40

#service libvirtd restart

2. concurrent starting 50 lxc containers
#for i in {1..50}; do virsh -c lxc:/// start lxc-test-$i & done

All the lxc containers are started, no error in libvirtd.log. And from the libvirtd.log, the 'max_anonymous_clients' works

# grep virNetServerAddClient /var/log/libvirtd.log2014-11-06 05:45:06.761+0000: 24051: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients
2014-11-06 05:45:06.768+0000: 24051: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients
2014-11-06 05:45:06.913+0000: 24051: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients
2014-11-06 05:45:06.916+0000: 24051: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients
2014-11-06 05:45:06.921+0000: 24051: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients

# grep virNetServerCheckLimits /var/log/libvirtd.log
2014-11-06 05:45:06.733+0000: 24229: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=1 nclients_max=40 nclients_unauth=0 nclients_unauth_max=2
2014-11-06 05:45:06.733+0000: 24229: debug : virNetServerCheckLimits:1083 : Re-enabling services
2014-11-06 05:45:06.762+0000: 24058: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=3 nclients_max=40 nclients_unauth=1 nclients_unauth_max=2
2014-11-06 05:45:06.762+0000: 24058: debug : virNetServerCheckLimits:1083 : Re-enabling services
2014-11-06 05:45:06.912+0000: 24058: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=4 nclients_max=40 nclients_unauth=1 nclients_unauth_max=2
Comment 11 errata-xmlrpc 2015-03-05 07:24:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html