Hide Forgot
+++ This bug was initially created as a clone of Bug #981729 +++ Description of problem: The 'max_clients' setting controls how many clients are allowed to connect to libvirt, as a protection against DOS attacker from unauthenticated users. It is problematic when dealing with concurrent start of large numbers of containers, because this can trigger a a very large number of connections in a short period of time. With the default max_clients=20, this easily causes container startup failure. We can't simply raise the limit since it is DOS protection, but we can improve the behaviour somewhat. Currently libvirt will unconditionally accept() any incoming socket, regardless of the current number of clients. Thus if the max limit is hit, libvirt will accept and immediately close client connections. It would be better if libvirt simply did not accept() the connection. This would let pending connections wait for a previous connection to close before continuing. The max number of queued connections can be controlled via the listen() syscall, and could be a fairly large number, since they consume minimal resources. This would be a new "max_queued_clients" setting in libvirtd.conf, perhaps as much as 1000 A new 'max_anonymous_clients' setting could limit only those connections which are accept()ed, but not yet authenticated. This could be fairly low (perhaps current 20) The existing 'max_clients' setting could then be used as a limit on total number of connections, and set to a far higher value (perhaps several 100 or more) Version-Release number of selected component (if applicable): 1.1.0-1.el6. How reproducible: Always Steps to Reproduce: 1. Attempt to open 30 concurrent connections to libvirt 2. 3. Actual results: Only first 20 succeed, the rest are dropped Expected results: 10 connections are queued, pending close of 10 earlier connections Additional info: --- Additional comment from RHEL Product and Program Management on 2013-07-05 17:47:50 CEST --- Since this bug report was entered in bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release. --- Additional comment from Alex Jia on 2013-07-08 12:23:37 CEST --- # tail -2 /etc/libvirt/libvirtd.conf max_clients = 20 max_workers = 20 # for i in {1..30}; do virt-sandbox-service create -C -u httpd.service -N dhcp myapache$i;done # for i in {1..30}; do virt-sandbox-service create start myapache$i & done XXX # Unable to open connection: Unable to open lxc:///: Cannot recv data: Connection reset by peer Unable to open connection: Unable to open lxc:///: Cannot recv data: Connection reset by peer Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe Unable to open connection: Unable to open lxc:///: Cannot write data: Broken pipe And check libvirtd log: 2013-07-08 10:13:58.933+0000: 8034: error : virNetServerAddClient:262 : Too many active clients (20), dropping connection from 127.0.0.1;0 2013-07-08 10:13:58.941+0000: 8034: error : virNetServerAddClient:262 : Too many active clients (20), dropping connection from 127.0.0.1;0 2013-07-08 10:13:58.943+0000: 8034: error : virNetServerAddClient:262 : Too many active clients (20), dropping connection from 127.0.0.1;0 --- Additional comment from Michal Privoznik on 2013-07-25 16:24:16 CEST --- I've just proposed patches upstream: https://www.redhat.com/archives/libvir-list/2013-July/msg01646.html --- Additional comment from Michal Privoznik on 2013-08-05 11:31:02 CEST --- Moving to POST: http://post-office.corp.redhat.com/archives/rhvirt-patches/2013-August/msg00079.html --- Additional comment from Daniel Berrange on 2013-08-05 11:58:33 CEST --- NB the upstream patches only implement half of this bug. There's still no separation of the limits for anonymous vs authenticated clients. This bug is to address the second part of original request = the 'max_anonymous_clients' setting.
Patch proposed upstream: https://www.redhat.com/archives/libvir-list/2013-December/msg00453.html
I've just pushed patches upstream: commit 68f60f669c566e53904ce39c95a57853f7f23638 Author: Michal Privoznik <mprivozn> AuthorDate: Tue Mar 4 18:55:24 2014 +0100 Commit: Michal Privoznik <mprivozn> CommitDate: Mon Mar 17 17:45:13 2014 +0100 daemon: Introduce max_anonymous_clients https://bugzilla.redhat.com/show_bug.cgi?id=992980 This config tunable allows users to determine the maximum number of accepted but yet not authenticated users. Signed-off-by: Michal Privoznik <mprivozn> commit 4015396b2cf9f43ff77d24e3a4d3e1372f5352a3 Author: Michal Privoznik <mprivozn> AuthorDate: Tue Mar 4 15:37:27 2014 +0100 Commit: Michal Privoznik <mprivozn> CommitDate: Mon Mar 17 17:37:42 2014 +0100 virNetServer: Introduce unauth clients counter The counter gets incremented on each unauthenticated client added to the server and decremented whenever the client authenticates. Signed-off-by: Michal Privoznik <mprivozn> v1.2.2-191-g68f60f6
Hi Michal, I set 'max_anonymous_clients' to '2' in libvirtd.conf. Then I tried to start 20 LXC containers. All of them are started. There is no error about 'dropping connection' in libvirt log. Could you please help check whether It can be verified by the following steps or not? Thanks in advance. Yang # rpm -q libvirt libvirt-1.2.8-5.el7.x86_64 # grep max /etc/libvirt/libvirtd.conf max_clients= 4 #max_queued_clients = 2 max_anonymous_clients = 2 max_workers =4 # virsh -c lxc:/// list --all Id Name State ---------------------------------------------------- - lxc-test-0 shut off - lxc-test-1 shut off - lxc-test-10 shut off - lxc-test-11 shut off - lxc-test-12 shut off - lxc-test-13 shut off - lxc-test-14 shut off - lxc-test-15 shut off - lxc-test-16 shut off - lxc-test-17 shut off - lxc-test-18 shut off - lxc-test-19 shut off - lxc-test-2 shut off - lxc-test-20 shut off - lxc-test-3 shut off - lxc-test-4 shut off - lxc-test-5 shut off - lxc-test-6 shut off - lxc-test-7 shut off - lxc-test-8 shut off - lxc-test-9 shut off # for i in {1..20}; do virsh -c lxc:/// start lxc-test-$i & done # virsh -c lxc:/// list Id Name State ---------------------------------------------------- 22004 lxc-test-13 running 22025 lxc-test-1 running 22031 lxc-test-7 running 22037 lxc-test-12 running 22049 lxc-test-5 running 22069 lxc-test-3 running 22078 lxc-test-18 running 22082 lxc-test-20 running 22102 lxc-test-16 running 22110 lxc-test-4 running 22113 lxc-test-11 running 22134 lxc-test-14 running 22142 lxc-test-6 running 22145 lxc-test-2 running 22156 lxc-test-17 running 22179 lxc-test-8 running 22186 lxc-test-9 running 22191 lxc-test-19 running 22202 lxc-test-15 running 22217 lxc-test-10 running [root@rhel7_test yy]# grep virNetServerAddClient /var/log/libvirtd.log 2014-10-30 08:40:49.111+0000: 20961: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients 2014-10-30 08:40:49.125+0000: 20961: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients 2014-10-30 08:40:49.127+0000: 20961: debug : virNetServerAddClient:295 : Temporarily suspending services due to max_clients 2014-10-30 08:41:19.346+0000: 20961: debug : virNetServerAddClient:295 : Temporarily suspending services due to max_clients 2014-10-30 08:41:49.574+0000: 20961: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients 2014-10-30 08:41:49.574+0000: 20961: debug : virNetServerAddClient:295 : Temporarily suspending services due to max_clients ...snip... [root@rhel7_test yy]# grep virNetServerCheckLimits /var/log/libvirtd.log 2014-10-30 08:40:49.113+0000: 20965: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=2 nclients_max=4 nclients_unauth=1 nclients_unauth_max=2 2014-10-30 08:40:49.113+0000: 20965: debug : virNetServerCheckLimits:1083 : Re-enabling services 2014-10-30 08:40:49.126+0000: 20970: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=3 nclients_max=4 nclients_unauth=1 nclients_unauth_max=2 2014-10-30 08:40:49.126+0000: 20970: debug : virNetServerCheckLimits:1083 : Re-enabling services 2014-10-30 08:40:49.127+0000: 20970: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=3 nclients_max=4 nclients_unauth=0 nclients_unauth_max=2 2014-10-30 08:40:49.127+0000: 20970: debug : virNetServerCheckLimits:1083 : Re-enabling services 2014-10-30 08:40:49.128+0000: 20968: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=4 nclients_max=4 nclients_unauth=0 nclients_unauth_max=2 2014-10-30 08:41:19.343+0000: 20961: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=3 nclients_max=4 nclients_unauth=0 nclients_unauth_max=2 2014-10-30 08:41:19.343+0000: 20961: debug : virNetServerCheckLimits:1083 : Re-enabling services ...snip...
(In reply to yangyang from comment #5) > Hi Michal, > > I set 'max_anonymous_clients' to '2' in libvirtd.conf. Then I tried to start > 20 LXC containers. All of them are started. There is no error about > 'dropping connection' in libvirt log. Could you please help check whether It > can be verified by the following steps or not? > > Thanks in advance. > Yang > > # rpm -q libvirt > libvirt-1.2.8-5.el7.x86_64 > > # grep max /etc/libvirt/libvirtd.conf > max_clients= 4 > #max_queued_clients = 2 > max_anonymous_clients = 2 > max_workers =4 This could be used, however, you need to set max_queued_clients=0. The problem is, even though libvirt doesn't accept incoming client on the socket, kernel will do parial opening, and queue clients on the socket from which they are taken off by calling accept(). The size of the queue is managed by max_queued_clients. So to disable this set it to zero and you should start seeing connection errors.
(In reply to Michal Privoznik from comment #6) > (In reply to yangyang from comment #5) > > Hi Michal, > > > > I set 'max_anonymous_clients' to '2' in libvirtd.conf. Then I tried to start > > 20 LXC containers. All of them are started. There is no error about > > 'dropping connection' in libvirt log. Could you please help check whether It > > can be verified by the following steps or not? > > > > Thanks in advance. > > Yang > > > > # rpm -q libvirt > > libvirt-1.2.8-5.el7.x86_64 > > > > # grep max /etc/libvirt/libvirtd.conf > > max_clients= 4 > > #max_queued_clients = 2 > > max_anonymous_clients = 2 > > max_workers =4 > > This could be used, however, you need to set max_queued_clients=0. The > problem is, even though libvirt doesn't accept incoming client on the > socket, kernel will do parial opening, and queue clients on the socket from > which they are taken off by calling accept(). The size of the queue is > managed by max_queued_clients. So to disable this set it to zero and you > should start seeing connection errors. If 'max_queued_clients' is set to zero, it will be translated into 30, right? If 'max_anonymous_clients' is set to '-1', it will be translated into '18446744073709551615', is it expected result ? for example: # grep max /etc/libvirt/libvirtd.conf max_clients= 40 max_queued_clients = 0 max_anonymous_clients = -1 max_workers =40 #grep virNetServerCheckLimits /var/log/libvirtd.log 2014-10-31 05:38:14.146+0000: 29136: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=0 nclients_max=40 nclients_unauth=0 nclients_unauth_max=18446744073709551615 2014-10-31 05:38:14.146+0000: 29136: debug : virNetServerCheckLimits:1083 : Re-enabling services Thanks Yang
(In reply to yangyang from comment #7) > (In reply to Michal Privoznik from comment #6) > > (In reply to yangyang from comment #5) > > > Hi Michal, > > > > > > I set 'max_anonymous_clients' to '2' in libvirtd.conf. Then I tried to start > > > 20 LXC containers. All of them are started. There is no error about > > > 'dropping connection' in libvirt log. Could you please help check whether It > > > can be verified by the following steps or not? > > > > > > Thanks in advance. > > > Yang > > > > > > # rpm -q libvirt > > > libvirt-1.2.8-5.el7.x86_64 > > > > > > # grep max /etc/libvirt/libvirtd.conf > > > max_clients= 4 > > > #max_queued_clients = 2 > > > max_anonymous_clients = 2 > > > max_workers =4 > > > > This could be used, however, you need to set max_queued_clients=0. The > > problem is, even though libvirt doesn't accept incoming client on the > > socket, kernel will do parial opening, and queue clients on the socket from > > which they are taken off by calling accept(). The size of the queue is > > managed by max_queued_clients. So to disable this set it to zero and you > > should start seeing connection errors. > > If 'max_queued_clients' is set to zero, it will be translated into 30, right? Correct. > > If 'max_anonymous_clients' is set to '-1', it will be translated into > '18446744073709551615', is it expected result ? Well, it's a broader problem I think. I mean, we don't check for negative values in other cases too (e.g. all these max_*). So I'd save it for separate bug. It shouldn't be a show stopper for this feature. > > for example: > # grep max /etc/libvirt/libvirtd.conf > max_clients= 40 > max_queued_clients = 0 > max_anonymous_clients = -1 > max_workers =40 > > #grep virNetServerCheckLimits /var/log/libvirtd.log > 2014-10-31 05:38:14.146+0000: 29136: debug : virNetServerCheckLimits:1078 : > Considering re-enabling services: nclients=0 nclients_max=40 > nclients_unauth=0 nclients_unauth_max=18446744073709551615 > 2014-10-31 05:38:14.146+0000: 29136: debug : virNetServerCheckLimits:1083 : > Re-enabling services > So it works, nice.
Opened a separate bug to track the negative value issue https://bugzilla.redhat.com/show_bug.cgi?id=1160995 Verify this one as following Product version libvirt-1.2.8-6.el7.x86_64 Steps 1. set 'max_*' as following # grep max /etc/libvirt/libvirtd.conf max_clients= 40 max_queued_clients = 0 max_anonymous_clients = 2 max_workers = 40 #service libvirtd restart 2. concurrent starting 50 lxc containers #for i in {1..50}; do virsh -c lxc:/// start lxc-test-$i & done All the lxc containers are started, no error in libvirtd.log. And from the libvirtd.log, the 'max_anonymous_clients' works # grep virNetServerAddClient /var/log/libvirtd.log2014-11-06 05:45:06.761+0000: 24051: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients 2014-11-06 05:45:06.768+0000: 24051: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients 2014-11-06 05:45:06.913+0000: 24051: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients 2014-11-06 05:45:06.916+0000: 24051: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients 2014-11-06 05:45:06.921+0000: 24051: debug : virNetServerAddClient:289 : Temporarily suspending services due to max_anonymous_clients # grep virNetServerCheckLimits /var/log/libvirtd.log 2014-11-06 05:45:06.733+0000: 24229: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=1 nclients_max=40 nclients_unauth=0 nclients_unauth_max=2 2014-11-06 05:45:06.733+0000: 24229: debug : virNetServerCheckLimits:1083 : Re-enabling services 2014-11-06 05:45:06.762+0000: 24058: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=3 nclients_max=40 nclients_unauth=1 nclients_unauth_max=2 2014-11-06 05:45:06.762+0000: 24058: debug : virNetServerCheckLimits:1083 : Re-enabling services 2014-11-06 05:45:06.912+0000: 24058: debug : virNetServerCheckLimits:1078 : Considering re-enabling services: nclients=4 nclients_max=40 nclients_unauth=1 nclients_unauth_max=2
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html