Bug 864078 - failover_soak sometimes highlights low-level client reconnect failures (exit code of 6)
failover_soak sometimes highlights low-level client reconnect failures (exit ...
Status: CLOSED WONTFIX
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
Development
Unspecified Unspecified
medium Severity medium
: 3.0
: ---
Assigned To: mick
MRG Quality Engineering
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-08 10:09 EDT by Frantisek Reznicek
Modified: 2015-11-15 20:14 EST (History)
4 users (show)

See Also:
Fixed In Version: 0.18
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-23 07:35:56 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
failover_soak and broker logs (215.58 KB, application/x-bzip2)
2012-10-08 10:14 EDT, Frantisek Reznicek
no flags Details

  None (edit)
Description Frantisek Reznicek 2012-10-08 10:09:57 EDT
Description of problem:
failover_soak highlights low-level client reconnect failures (exit code of 6).

Test failover_soak sometimes (seen on RHEL 5.8 / 6.3 i686) fails with 
END_OF_TEST ERROR_CLIENT (101/0)

It looks that low-level clients in 32 bit environment fails to reconnect...

failover_soak logging:

  soak-0: 2012-10-08 06:03:42 notice cluster(10.16.66.66:29085 READY) joined cluster soakTestCluster_cc3f8332-e3fc-4e6b-b6b9-431a3f9a83aa
  new broker created: pid == 29085 log-prefix == soak-0
  soak-0: 2012-10-08 06:03:42 notice Broker running
  ...
  new broker created: pid == 29090 log-prefix == soak-1
  Broker List ------------ size: 2
  pid: 29085   port: 41347
  pid: 29090   port: 49818
  end Broker List ------------
  2012-10-08_06:03:43 Declaring queues...
  ...
  2012-10-08_06:08:05 Killing oldest broker...
  killFrontBroker: waiting for newbie sync...
  killFrontBroker: newbie synced.
  killFrontBroker pid: 29085 on port 41347 using signal 9
  ...
  new broker created: pid == 29122 log-prefix == soak-2
  Broker List ------------ size: 2
  pid: 29090   port: 49818
  pid: 29122   port: 49797
  end Broker List ------------
  2012-10-08_06:09:08 1 brokers have been killed.
  ...
  new broker created: pid == 29137 log-prefix == soak-3
  Broker List ------------ size: 2
  pid: 29122   port: 49797
  pid: 29137   port: 43999
  end Broker List ------------
  2012-10-08_06:10:02 2 brokers have been killed.
  ...
  new broker created: pid == 29155 log-prefix == soak-4
  Broker List ------------ size: 2
  pid: 29137   port: 43999
  pid: 29155   port: 45541
  end Broker List ------------
  2012-10-08_06:15:12 3 brokers have been killed.
  ...
  new broker created: pid == 29170 log-prefix == soak-5
  Broker List ------------ size: 2
  pid: 29155   port: 45541
  pid: 29170   port: 38701
  end Broker List ------------
  2012-10-08_06:16:09 4 brokers have been killed.
  ...
  new broker created: pid == 29185 log-prefix == soak-6
  Broker List ------------ size: 2
  pid: 29170   port: 38701
  pid: 29185   port: 46375
  end Broker List ------------
  2012-10-08_06:17:14 5 brokers have been killed.
  ...
  new broker created: pid == 29200 log-prefix == soak-7
  Broker List ------------ size: 2
  pid: 29185   port: 46375
  pid: 29200   port: 51957
  end Broker List ------------
  2012-10-08_06:20:31 6 brokers have been killed.
  ...
  new broker created: pid == 29214 log-prefix == soak-8
  Broker List ------------ size: 2
  pid: 29200   port: 51957
  pid: 29214   port: 59325
  end Broker List ------------
  2012-10-08_06:20:59 7 brokers have been killed.

Sender logging:

  SENDER started (PID:29105)
  2012-10-08 06:03:55 info ...
  ...
  SENDER (host: 127.0.0.1 port: 41347 )  PID: 29105 Failed: Cannot establish a connection
  2012-10-08 06:20:58 info ...

Broker does not show any suspicious messages.


Version-Release number of selected component (if applicable):
 qpid-cpp-client-0.14-22.el6_3.i686
 qpid-cpp-client-devel-0.14-22.el6_3.i686
 qpid-cpp-client-devel-docs-0.14-22.el6_3.noarch
 qpid-cpp-client-rdma-0.14-22.el6_3.i686
 qpid-cpp-client-ssl-0.14-22.el6_3.i686
 qpid-cpp-debuginfo-0.14-22.el6_3.i686
 qpid-cpp-server-0.14-22.el6_3.i686
 qpid-cpp-server-cluster-0.14-22.el6_3.i686
 qpid-cpp-server-devel-0.14-22.el6_3.i686
 qpid-cpp-server-rdma-0.14-22.el6_3.i686
 qpid-cpp-server-ssl-0.14-22.el6_3.i686
 qpid-cpp-server-store-0.14-22.el6_3.i686
 qpid-cpp-server-xml-0.14-22.el6_3.i686
 qpid-java-client-0.18-2.el6.noarch
 qpid-java-common-0.18-2.el6.noarch
 qpid-java-example-0.18-2.el6.noarch
 qpid-qmf-0.14-14.el6_3.i686
 qpid-qmf-debuginfo-0.14-14.el6_3.i686
 qpid-qmf-devel-0.14-14.el6_3.i686
 qpid-tests-0.14-1.el6_2.noarch
 qpid-tools-0.14-6.el6_3.noarch
 python-qpid-0.14-11.el6_3.noarch
 python-qpid-qmf-0.14-14.el6_3.i686


How reproducible:
5%

Steps to Reproduce:
1. run failover_soak in loop as qpid_ptest_cluster_failover_soak does
2. watch for exit codes == 6
  
Actual results:
failover_soak fails.

Expected results:
failover_soak should pass.

Additional info:
Comment 2 Frantisek Reznicek 2012-10-08 10:14:21 EDT
Created attachment 623470 [details]
failover_soak and broker logs
Comment 4 Frantisek Reznicek 2012-10-23 11:03:11 EDT
Frantisek, retest on 0.18.
Comment 5 Justin Ross 2012-10-24 16:02:25 EDT
This is medium (or perhaps low) priority because it affects only the old client api, not the new messaging api.  Moving out to the next release.
Comment 6 Frantisek Reznicek 2012-11-14 04:22:31 EST
The current runs on 0.18 based qpid show that issue is not occurring anymore, dependency to bug 875660. Will retest workarounding bug 875660 or eventually after bug 875660 is fixed.
Comment 7 Justin Ross 2013-02-22 07:41:38 EST
(In reply to comment #6)
> The current runs on 0.18 based qpid show that issue is not occurring
> anymore, dependency to bug 875660. Will retest workarounding bug 875660 or
> eventually after bug 875660 is fixed.

Okay, moving this to modified for now.
Comment 8 Leonid Zhaldybin 2013-02-22 08:18:18 EST
The bug 875660 has been verified, but we still see the initial issue occurring sporadically on the latest 0.18 builds.

-> ASSIGNED
Comment 9 Justin Ross 2013-02-23 07:35:56 EST
Since this is a bug against the deprecated client api, -> wontfix

Note You need to log in before you can comment on or make changes to this bug.