Bug 812244 - clustered qpidd in auth=yes mode gets unavailable as runs out of available connections (within few hours due to default sesame)
clustered qpidd in auth=yes mode gets unavailable as runs out of available co...
Status: CLOSED DUPLICATE of bug 729311
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
Development
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: mick
MRG Quality Engineering
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-04-13 04:16 EDT by Frantisek Reznicek
Modified: 2015-11-15 20:14 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-17 13:15:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Logs and configurations (13.28 KB, application/x-tbz)
2012-04-13 04:20 EDT, Frantisek Reznicek
no flags Details

  None (edit)
Description Frantisek Reznicek 2012-04-13 04:16:05 EDT
Description of problem:

Clustered qpidd in auth=yes mode gets unavailable as runs out of available connections (within few hours due to default sesame)

Situation:
Two clustered Qpidd brokers in auth=yes mode, when authentication mechanism is forced to be PLAIN only (at sasl level)

  # cat /etc/qpidd.conf
  cluster-mechanism=DIGEST-MD5 ANONYMOUS PLAIN
  log-enable=info+
  log-enable=debug+:msgstore
  mgmt-pub-interval=5
  log-to-file=/var/lib/qpidd/qpidd.log
  cluster-name=mycluster_10.34.27.49_10.34.27.50
  auth=yes
  cluster-username=guest
  cluster-password=guest


  # tail -5 /etc/sasl2/qpidd.conf
  #cyrus-sql-sasl plugin is installed
  sql_select: dummy select
  
  
  mech_list: PLAIN

In following configuration cluster starts and works as expected.

Both nodes run also defaultly configured sesame daemon which is trying to connect to qpidd using ANONYMOUS mechanism. (which is not allowed, see /etc/sasl2/qpidd.conf above)

No other qpidd clients are active.

After couple of hours cluster become unavailable, both qpidd processes are up, but rejecting any incoming connections:

  # qpid-stat -b --sasl-mechanism=PLAIN guest/guest@localhost
  Failed: ConnectionFailed - (None, 'connection aborted')


Looking in more detail to qpidd log I found:

  2012-04-13 09:08:05 info SASL: Mechanism list: PLAIN
  2012-04-13 09:08:05 warning Client closed connection with 501: Desired mechanism(s) not valid: ANONYMOUS (supported: PLAIN) (qpid/client/Conne
  ctionHandler.cpp:252)
  ...
  2012-04-13 09:16:38 info SASL: Mechanism list: PLAIN
  2012-04-13 09:16:38 warning Client closed connection with 501: Desired mechanism(s) not valid: ANONYMOUS (supported: PLAIN) (qpid/client/Conne
  ctionHandler.cpp:252)
  2012-04-13 09:18:46 info SASL: Mechanism list: PLAIN
  2012-04-13 09:18:46 warning Client closed connection with 501: Desired mechanism(s) not valid: ANONYMOUS (supported: PLAIN) (qpid/client/Conne
  ctionHandler.cpp:252)
  2012-04-13 09:20:54 error Client max connection count limit exceeded: 500 connection refused
  2012-04-13 09:23:02 error Client max connection count limit exceeded: 500 connection refused
  2012-04-13 09:23:47 error Client max connection count limit exceeded: 500 connection refused
  ...


It is evident that two sesame instances (only qpidd clients) connections are not refused properly.

At this point it is impossible to debug what connections and/or broker content is stored.

Issue seen in rhel6.2 i686+x86_64 cluster.

Version-Release number of selected component (if applicable):
  python-qpid-0.14-7.el6_2
  python-qpid-qmf-0.14-7.el6_2
  python-saslwrapper-0.10-2.el6
  qpid-cpp-*-0.14-14.el6_2
  qpid-java-*-0.14-3.el6
  qpid-qmf-*0.14-7.el6_2
  qpid-tests-0.14-1.el6_2
  qpid-tools-0.14-2.el6_2
  rh-qpid-cpp-tests-0.14-14.el6_2
  ruby-qpid-qmf-0.14-7.el6_2
  ruby-saslwrapper-0.10-2.el6
  saslwrapper-*0.10-2.el6
  sesame-1.0-5.el6


How reproducible:
not available, one attempt so far

Steps to Reproduce:
1. set-up 2 cluster to be in auth=yes
2. start cluster
3. start default sesame
4. qpid-stat -b --sasl-mechanism=PLAIN guest/guest@localhost
5. sleep 10h
6. qpid-stat -b --sasl-mechanism=PLAIN guest/guest@localhost
  
Actual results:
Cluster become unavailable for clients.

Expected results:
Cluster should be available for clients.

Additional info:
Comment 1 Frantisek Reznicek 2012-04-13 04:20:00 EDT
Created attachment 577266 [details]
Logs and configurations


[root@dhcp-27-49 ~]# cat /etc/qpidd.conf
cluster-mechanism=DIGEST-MD5 ANONYMOUS PLAIN
log-enable=info+
log-enable=debug+:msgstore
mgmt-pub-interval=5
log-to-file=/var/lib/qpidd/qpidd.log
cluster-name=mycluster_10.34.27.49_10.34.27.50
auth=yes
cluster-username=guest
cluster-password=guest
[root@dhcp-27-49 ~]# service sesame status
sesame (pid 1146) is running...
[root@dhcp-27-49 ~]# tail -2 /etc/sasl2/qpidd.conf
mech_list: PLAIN


[root@dhcp-27-50 ~]# cat /etc/qpidd.conf
cluster-mechanism=DIGEST-MD5 ANONYMOUS PLAIN
log-enable=info+
log-enable=debug+:msgstore
mgmt-pub-interval=5
log-to-file=/var/lib/qpidd/qpidd.log
cluster-name=mycluster_10.34.27.49_10.34.27.50
auth=yes
cluster-username=guest
cluster-password=guest
[root@dhcp-27-50 ~]# service sesame status
sesame (pid 1174) is running...
[root@dhcp-27-50 ~]# tail -2 /etc/sasl2/qpidd.conf
mech_list: PLAIN


[root@dhcp-27-49 ~]# qpid-stat -b --sasl-mechanism=PLAIN guest/guest@localhost
Failed: ConnectionFailed - (None, 'connection aborted')

[root@dhcp-27-50 ~]# qpid-stat -b --sasl-mechanism=PLAIN guest/guest@localhost
Failed: ConnectionFailed - (None, 'connection aborted')


Full logs and configs in attachment.
Comment 4 Frantisek Reznicek 2013-03-18 11:04:32 EDT
Yes, I believe that this defect is dup of Bug 729311.

Connection object was not deleted and after 500 connections client / sesame got 'error Client max connection count limit exceeded: 500 connection refused', which is compatible with Gordon's bug 729311, comment 1.
Comment 5 Justin Ross 2014-06-17 13:15:23 EDT

*** This bug has been marked as a duplicate of bug 729311 ***

Note You need to log in before you can comment on or make changes to this bug.