Bug 855403

Summary: WinSDK SSL client: memory leak; server exit hang
Product: Red Hat Enterprise MRG Reporter: Chuck Rolke <crolke>
Component: qpid-sdkAssignee: Chuck Rolke <crolke>
Status: CLOSED CURRENTRELEASE QA Contact: Petra Svobodová <psvobodo>
Severity: high Docs Contact:
Priority: high    
Version: 2.2CC: jross, lzhaldyb, mcressma, psvobodo
Target Milestone: 2.3   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qpid-cpp-win-3.2.0.2 Doc Type: Bug Fix
Doc Text:
Cause: Code in Windows SslConnector failed to pass connection status to common IO modules in Windows client. Consequence: Connection structures were never cleaned up properly and memory consumption grew as connections came and went. Users of fetch() library function would hang if the server closed the connection to the client. Fix: Pass connection status to common IO modules. Result: Memory consumption and code that called fetch() behave normally.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-03-19 16:37:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Example source code to demonstrate the two errors.
none
bz855403_ssl_mem_leak.cpp none

Description Chuck Rolke 2012-09-07 15:58:30 UTC
Description of problem:

Using SSL connections with the WinSDK leaks memory.
Using SSL connections with the WinSDK a client program that executes a fetch() with timeout FOREVER will hang if the broker exits.


Version-Release number of selected component (if applicable):

3.1.0.2 (tag qpid-cpp-0.14-20)


How reproducible:

100%

Steps to Reproduce:
1. Modify hello_world.cpp, native C++, to repeatedly send and receive its message.
2. launch with: hello_world ssl-server-address amq.topic "{transport: 'ssl'}"
3. monitor the hello_world.exe and watch private memory bytes expand.
  
Actual results:

Client executable grows at ~300k bytes per connection. 

Expected results:

Client executable should have constant memory usage.

Additional info:

Comment 1 Chuck Rolke 2012-09-07 16:02:46 UTC
To reproduce the hang:

1. Modify hello_world.cpp, native C++, to NOT send a message to the server but to then fetch with a FOREVER timeout.
2. launch with: hello_world ssl-server-address amq.topic "{transport: 'ssl'}"
3. Kill the connection to the broker or just have the broker exit.

Actual results:

Client stays in the fetch() call.

Expected results:

Client should continue execting when fetch() returns.

Comment 2 Chuck Rolke 2012-09-10 14:52:37 UTC
Created attachment 611451 [details]
Example source code to demonstrate the two errors.

This triggers the errors equally going to a Linux SSL broker or to a Windows SSL broker.

Comment 3 Chuck Rolke 2012-09-10 14:57:17 UTC
Fixed upstream with checkin r1382026

Comment 5 Petra Svobodová 2012-11-20 09:40:26 UTC
I tried to verify the memory leak issue. I used the "bz855403_ssl_mem_leak" example application, what is a slightly modified original attached example.

Steps:
1. I ran the "bz855403_ssl_mem_leak" application on the background from python script
2. In the marked points I read current memory usage of the process

Results on the qpid-cpp-win-3.2.0.2-1:

Iteration number       Current memory usage [KB]
1                      4632 
2                      8496
...                    ...
150                    9416 (


Results on the qpid-cpp-3.1.0.2-1 (previous version):

Iteration number       Current memory usage [KB]
1                      4620
2                      8456
...                    ...
150                    48360    

The measurements were executed on Windows7-x64 (client) and Rhel5-i386 (broker); the example was compiled in MS Visual Studio 2008.

The memory usage grows only slow in case the current packages, but the increment is evident.

Could you possibly evaluate if the memory usage increment is acceptable, please?

--> ASSIGNED

Comment 6 Petra Svobodová 2012-11-20 09:41:36 UTC
Created attachment 648363 [details]
bz855403_ssl_mem_leak.cpp

Comment 7 Justin Ross 2012-11-20 18:45:12 UTC
That increase is small enough that it could be the result of memory fragmentation.  In order to tell, you'd need to run the test over a long period, on the order of 24 hours.  With enough time, assuming there's not a leak, the memory usage should stabilize.  Chuck, thoughts?

Comment 8 Chuck Rolke 2012-11-20 19:26:32 UTC
There is still a leak of about 1k per connection. 
See https://issues.apache.org/jira/browse/QPID-4295

Comment 9 Justin Ross 2012-11-20 19:29:09 UTC
Okay, let's move this one to MODIFIED and create a new bz for QPID-4295.

(In reply to comment #8)
> There is still a leak of about 1k per connection. 
> See https://issues.apache.org/jira/browse/QPID-4295

Comment 10 Petra Svobodová 2012-11-26 08:52:38 UTC
After broker exit the client tries to reconnect the broker now; no hang appeared.

Verified on packages qpid-cpp-win-3.2.0.3-1 and qpid-cpp-0.18-10 (on the broker side) on WindowsXP-x86 and x64, Windows7-x86 and x64, Windows Server2003-x86 and x64, Winows Server2008-x86, x64 and R2.

Could you possibly add a link to the new bugzilla (with the hang part), please? I would like to close this one, but I cannot do it without the new one. Thank you in advance.

Comment 11 Chuck Rolke 2012-11-26 15:22:35 UTC
The new bugzilla is https://bugzilla.redhat.com/show_bug.cgi?id=878646

However, it addresses a memory leak. The hang is contained to this BZ.

Comment 12 Petra Svobodová 2012-11-26 15:48:33 UTC
The first issue "using SSL connections with the WinSDK leaks memory" is solved in the bz 878646; see the previous comment, please.


The second issue "Using SSL connections with the WinSDK a client program that executes a fetch() with timeout FOREVER will hang if the broker exits." did not appear yet; after broker exit the client tries to reconnect the broker repeatedly.

Verified on packages qpid-cpp-win-3.2.0.3-1 and qpid-cpp-0.18-10 (on the broker side) on WindowsXP-x86 and x64, Windows7-x86 and x64, Windows Server2003-x86 and x64, Winows Server2008-x86, x64 and R2.

--> VERIFIED