Description of problem:

There are multiple observations that reconnecting sender (csharp.example.spout, with bug 657523 c#2 and bug 629926 c#9 fixes)

fails to reconnect with following exception:
Exception Org.Apache.Qpid.Messaging.QpidException: Failed to connect within reconnect timeout
   at Org.Apache.Qpid.Messaging.Sender.Send(Message mmsgp, Boolean sync)
   at Org.Apache.Qpid.Messaging.Sender.Send(Message mmsgp)
   at Org.Apache.Qpid.Messaging.Examples.Spout.Main(String[] args) in c:\qpid\do
tnet_examples\examples\csharp.example.spout\csharp.example.spout.cs:line 96.

Moreover at the bad reconnect point I can see that messages in the queue are purged, which seems to be another broker related problem.

Seen on Win XP / Server 2003 x64 vs. RHEL4.8 i386.


Version-Release number of selected component (if applicable):
qpid-cpp-winsdk-1.3.0.24-*.zip

How reproducible:
40%

Steps to Reproduce:
1. extract qpid winsdk
2. patch csharp.example.spout .NET example (bug 657523 c#2 and bug 629926 c#9 fixes)
3. compile the examples
4. execute sender with following config
...\csharp.example.spout.exe --broker <host>:<port> 
  --connection-options "{reconnect : True, reconnect-timeout : 2, reconnect_limit : 10, reconnect_interval : 3, username: guest, password:guest}"
  --content "<random-string>" --count <random-number-gt-1000> 
  "<rand-str-addr>; { create:sender, delete: receiver, node: { type:queue, durable:True } }"
5. while spout is running restart the broker on <host> 'service qpidd restart'
6. monitor queue depth by qc_list_objects.py or qpid-tool to see depth
   check csharp.example.spout.exe messages
7. Repeat point 4. - 6. until you see 
   a] Reconnection failed as above
   b] queue depth dropped

See more details in last section...

Actual results:
.NET client reconnection function is not reliable, it sometimes fails and messages might be lost

Expected results:
.NET client reconnection function should be reliable, no message loss should happen

Additional info:

  
  Reconnection issue description:
  
  C:\qpid>c:\qpid\bin\csharp.example.spout.exe --broker mrg-qe-07.lab.eng.brq.redh
  at.com:5672 --connection-options "{reconnect : True, reconnect-timeout : 2, reco
  nnect_limit : 10, reconnect_interval : 3, username: guest, password:guest}" --co
  ntent 1w11N67MJspuIFVzuIk82BCMI5v8gXCNZZ3q4DUn --count 1004 "VPO3E25FnRz2FBx3Qk8
  V4czEah44aTu3OZe2; { create:sender, delete: receiver, node: { type:queue, durabl
  e:True } }"
  
  1> service qpidd restart  # reconnection triggered
  
  2010-12-02 15:22:51 warning Connection [2298 10.34.33.60:5672] closed
  2010-12-02 15:22:52 warning Connection [2299 10.34.33.60:5672] closed
  2010-12-02 15:22:53 warning Connect failed: No connection could be made because
  the target machine actively refused it. :  (..\..\qpid-cpp-mrg-0.7.946106\cpp\sr
  c\qpid\sys\windows\Socket.cpp:219)
  2010-12-02 15:22:53 warning Connection  closed
  
  >> reconnection succeeded
  >> queue contains:
  >> ./qc_list_objects.py -b mrg-qe-07 -qv | grep VPO3E25FnRz2FBx3Qk8V4czEah44aTu3OZe
  >> VPO3E25FnRz2FBx3Qk8V4czEah44aTu3OZe2  durable msg|byteEnq/Deq:1004/0|40160/0 unacked:0 consumer_cnt:0 binding_cnt:1
  
  
  C:\qpid>c:\qpid\bin\csharp.example.spout.exe --broker mrg-qe-07.lab.eng.brq.redh
  at.com:5672 --connection-options "{reconnect : True, reconnect-timeout : 2, reco
  nnect_limit : 10, reconnect_interval : 3, username: guest, password:guest}" --co
  ntent 1w11N67MJspuIFVzuIk82BCMI5v8gXCNZZ3q4DUn --count 1004 "VPO3E25FnRz2FBx3Qk8
  V4czEah44aTu3OZe2; { create:sender, delete: receiver, node: { type:queue, durabl
  e:True } }"
  
  1> service qpidd restart  # but late, reconnection code was not triggered
  
  >> no reconnection happened
  >> queue contains:
  >> ./qc_list_objects.py -b mrg-qe-07 -qv | grep VPO3E25FnRz2FBx3Qk8V4czEah44aTu3OZe
  >> VPO3E25FnRz2FBx3Qk8V4czEah44aTu3OZe2  durable msg|byteEnq/Deq:2008/0|80320/0 unacked:0 consumer_cnt:0 binding_cnt:1
  
  
  C:\qpid>c:\qpid\bin\csharp.example.spout.exe --broker mrg-qe-07.lab.eng.brq.redh
  at.com:5672 --connection-options "{reconnect : True, reconnect-timeout : 2, reco
  nnect_limit : 10, reconnect_interval : 3, username: guest, password:guest}" --co
  ntent 1w11N67MJspuIFVzuIk82BCMI5v8gXCNZZ3q4DUn --count 1004 "VPO3E25FnRz2FBx3Qk8
  V4czEah44aTu3OZe2; { create:sender, delete: receiver, node: { type:queue, durabl
  e:True } }"
  
  1> service qpidd restart  # reconnection triggered
  
  2010-12-02 15:23:34 warning Connection [2303 10.34.33.60:5672] closed
  2010-12-02 15:23:34 warning Connection [2304 10.34.33.60:5672] closed
  2010-12-02 15:23:36 warning Connect failed: No connection could be made because
  the target machine actively refused it. :  (..\..\qpid-cpp-mrg-0.7.946106\cpp\sr
  c\qpid\sys\windows\Socket.cpp:219)
  2010-12-02 15:23:36 warning Connection  closed
  Exception Org.Apache.Qpid.Messaging.QpidException: Failed to connect within reco
  nnect timeout
    at Org.Apache.Qpid.Messaging.Sender.Send(Message mmsgp, Boolean sync)
    at Org.Apache.Qpid.Messaging.Sender.Send(Message mmsgp)
    at Org.Apache.Qpid.Messaging.Examples.Spout.Main(String[] args) in c:\qpid\do
  tnet_examples\examples\csharp.example.spout\csharp.example.spout.cs:line 96.
  
  
  >> csharp.example.spout.exe ends with qpid exception and does not try again to reconnect -> ISSUE alpha]
  >> queue contains:
  >> ./qc_list_objects.py -b mrg-qe-07 -qv | grep VPO3E25FnRz2FBx3Qk8V4czEah44aTu3OZe
  >> VPO3E25FnRz2FBx3Qk8V4czEah44aTu3OZe2  durable msg|byteEnq/Deq:0/0|0/0 unacked:0 consumer_cnt:0 binding_cnt:1
  >> confirmed by qpid-tool, no message in the queue means -> ISSUE beta] MESSAGE LOSS
  
  
  Legend:
  C:\qpid> is on Win XP / Server 2003 x64
  1> is on RHEL 4.8 i386
  >> comments