Description of problem: Clustered qpidd in auth=yes mode gets unavailable as runs out of available connections (within few hours due to default sesame) Situation: Two clustered Qpidd brokers in auth=yes mode, when authentication mechanism is forced to be PLAIN only (at sasl level) # cat /etc/qpidd.conf cluster-mechanism=DIGEST-MD5 ANONYMOUS PLAIN log-enable=info+ log-enable=debug+:msgstore mgmt-pub-interval=5 log-to-file=/var/lib/qpidd/qpidd.log cluster-name=mycluster_10.34.27.49_10.34.27.50 auth=yes cluster-username=guest cluster-password=guest # tail -5 /etc/sasl2/qpidd.conf #cyrus-sql-sasl plugin is installed sql_select: dummy select mech_list: PLAIN In following configuration cluster starts and works as expected. Both nodes run also defaultly configured sesame daemon which is trying to connect to qpidd using ANONYMOUS mechanism. (which is not allowed, see /etc/sasl2/qpidd.conf above) No other qpidd clients are active. After couple of hours cluster become unavailable, both qpidd processes are up, but rejecting any incoming connections: # qpid-stat -b --sasl-mechanism=PLAIN guest/guest@localhost Failed: ConnectionFailed - (None, 'connection aborted') Looking in more detail to qpidd log I found: 2012-04-13 09:08:05 info SASL: Mechanism list: PLAIN 2012-04-13 09:08:05 warning Client closed connection with 501: Desired mechanism(s) not valid: ANONYMOUS (supported: PLAIN) (qpid/client/Conne ctionHandler.cpp:252) ... 2012-04-13 09:16:38 info SASL: Mechanism list: PLAIN 2012-04-13 09:16:38 warning Client closed connection with 501: Desired mechanism(s) not valid: ANONYMOUS (supported: PLAIN) (qpid/client/Conne ctionHandler.cpp:252) 2012-04-13 09:18:46 info SASL: Mechanism list: PLAIN 2012-04-13 09:18:46 warning Client closed connection with 501: Desired mechanism(s) not valid: ANONYMOUS (supported: PLAIN) (qpid/client/Conne ctionHandler.cpp:252) 2012-04-13 09:20:54 error Client max connection count limit exceeded: 500 connection refused 2012-04-13 09:23:02 error Client max connection count limit exceeded: 500 connection refused 2012-04-13 09:23:47 error Client max connection count limit exceeded: 500 connection refused ... It is evident that two sesame instances (only qpidd clients) connections are not refused properly. At this point it is impossible to debug what connections and/or broker content is stored. Issue seen in rhel6.2 i686+x86_64 cluster. Version-Release number of selected component (if applicable): python-qpid-0.14-7.el6_2 python-qpid-qmf-0.14-7.el6_2 python-saslwrapper-0.10-2.el6 qpid-cpp-*-0.14-14.el6_2 qpid-java-*-0.14-3.el6 qpid-qmf-*0.14-7.el6_2 qpid-tests-0.14-1.el6_2 qpid-tools-0.14-2.el6_2 rh-qpid-cpp-tests-0.14-14.el6_2 ruby-qpid-qmf-0.14-7.el6_2 ruby-saslwrapper-0.10-2.el6 saslwrapper-*0.10-2.el6 sesame-1.0-5.el6 How reproducible: not available, one attempt so far Steps to Reproduce: 1. set-up 2 cluster to be in auth=yes 2. start cluster 3. start default sesame 4. qpid-stat -b --sasl-mechanism=PLAIN guest/guest@localhost 5. sleep 10h 6. qpid-stat -b --sasl-mechanism=PLAIN guest/guest@localhost Actual results: Cluster become unavailable for clients. Expected results: Cluster should be available for clients. Additional info:
Created attachment 577266 [details] Logs and configurations [root@dhcp-27-49 ~]# cat /etc/qpidd.conf cluster-mechanism=DIGEST-MD5 ANONYMOUS PLAIN log-enable=info+ log-enable=debug+:msgstore mgmt-pub-interval=5 log-to-file=/var/lib/qpidd/qpidd.log cluster-name=mycluster_10.34.27.49_10.34.27.50 auth=yes cluster-username=guest cluster-password=guest [root@dhcp-27-49 ~]# service sesame status sesame (pid 1146) is running... [root@dhcp-27-49 ~]# tail -2 /etc/sasl2/qpidd.conf mech_list: PLAIN [root@dhcp-27-50 ~]# cat /etc/qpidd.conf cluster-mechanism=DIGEST-MD5 ANONYMOUS PLAIN log-enable=info+ log-enable=debug+:msgstore mgmt-pub-interval=5 log-to-file=/var/lib/qpidd/qpidd.log cluster-name=mycluster_10.34.27.49_10.34.27.50 auth=yes cluster-username=guest cluster-password=guest [root@dhcp-27-50 ~]# service sesame status sesame (pid 1174) is running... [root@dhcp-27-50 ~]# tail -2 /etc/sasl2/qpidd.conf mech_list: PLAIN [root@dhcp-27-49 ~]# qpid-stat -b --sasl-mechanism=PLAIN guest/guest@localhost Failed: ConnectionFailed - (None, 'connection aborted') [root@dhcp-27-50 ~]# qpid-stat -b --sasl-mechanism=PLAIN guest/guest@localhost Failed: ConnectionFailed - (None, 'connection aborted') Full logs and configs in attachment.
Yes, I believe that this defect is dup of Bug 729311. Connection object was not deleted and after 500 connections client / sesame got 'error Client max connection count limit exceeded: 500 connection refused', which is compatible with Gordon's bug 729311, comment 1.
*** This bug has been marked as a duplicate of bug 729311 ***