Bug 756790 - broker std::bad_alloc error
Summary: broker std::bad_alloc error
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: 2.1
Hardware: i686
OS: Unspecified
medium
high
Target Milestone: ---
: ---
Assignee: Andrew Stitcher
QA Contact: Leonid Zhaldybin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-11-24 14:53 UTC by Leonid Zhaldybin
Modified: 2014-11-09 22:38 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-11-06 18:53:24 UTC
Target Upstream Version:


Attachments (Terms of Use)
Script to reproduce the bug. (30.00 KB, application/x-tar)
2011-11-24 14:54 UTC, Leonid Zhaldybin
no flags Details
qpid broker log with std::bad_alloc error. (10.17 KB, application/x-bzip)
2011-11-24 14:58 UTC, Leonid Zhaldybin
no flags Details

Description Leonid Zhaldybin 2011-11-24 14:53:23 UTC
Description of problem:
If the 32-bit qpid broker has a queue with ~3GB of messages in it, trying to send something more to this queue results in std::bad_alloc error from broker. The client returns the connection error:
warning Connection [127.0.0.1:59573-127.0.0.1:5672] closed
Failed to connect (reconnect disabled)
It seems that 32-bit broker is unable to allocate memory somewhere above 3GB. 

Version-Release number of selected component (if applicable):
qpid-cpp-client-0.12-6.el6.i686
qpid-cpp-client-devel-0.12-6.el6.i686
qpid-cpp-client-rdma-0.12-6.el6.i686
qpid-cpp-client-ssl-0.12-6.el6.i686
qpid-cpp-debuginfo-0.12-6.el6.i686
qpid-cpp-server-0.12-6.el6.i686
qpid-cpp-server-cluster-0.12-6.el6.i686
qpid-cpp-server-rdma-0.12-6.el6.i686
qpid-cpp-server-ssl-0.12-6.el6.i686
qpid-cpp-server-store-0.12-6.el6.i68

How reproducible:
always


Steps to Reproduce:
1. Start up the 32-bit broker, create a queue with capacity more than 3GB.
2. Start a client which sends lots and lots of messages to this queue.
3. When broker's memory allocation reaches 3GB, it will unable to receive any more messages.
  
Actual results:
Broker throws std::bad_alloc error and does not accept any more messages.

Expected results:
Broker should be able to accept a message from a client in case that there is some free memory left on the machine and the target queue did not reach its capacity limit.


Additional info:
I tried to run a few test scenarios, such as:
1. Create one queue with a big capacity (4-5 GB). Try to send about 3.5 GB of messages to it. Messages were of a various sizes, from 1KB up to 100MB.
2. Create a number of queues (from 2 to 10) with big capacity. Try to distribute about 3.5 GB of messages among them.
The result was always the same: as soon as broker's memory consumption reached some value around 3GB, it was unable to receive messages any more.
So, either there is a bug in qpid broker, or there is a limitation of its capacity which cannot be overrun in 32-bit environment, in which case such a limitation should be properly described in user documentation.

Comment 1 Leonid Zhaldybin 2011-11-24 14:54:34 UTC
Created attachment 535946 [details]
Script to reproduce the bug.

Just untar it, change to bug_bad_alloc directory and type "make run".

Comment 2 Leonid Zhaldybin 2011-11-24 14:58:30 UTC
Created attachment 535947 [details]
qpid broker log with std::bad_alloc error.

Comment 3 Justin Ross 2011-11-28 14:56:31 UTC
This is in part fundamental; it's exhausting process virtual memory.  Is there a better way to handle it?

I'm inclined to seek a better log message (if possible) and a documentation fix.

Comment 4 Andrew Stitcher 2012-04-24 18:36:24 UTC
I concur with Justin - I don't think we can do much in the case that std::bad_alloc is thrown - it does mean that we can allocate no more memory. And in this case it almost certainly that we've run out of virtual address space.

We might be able to produce a log message before exiting, but we might not. The log subsystem itself needs to allocate memory.

I think that this can only be solved with documentation.

We may need to figure out safe maxima for the memory size that can be used.

Comment 6 Andrew Stitcher 2012-07-31 14:09:38 UTC
I don't think this is an actual bug, I propose closing it "won't fix."


Note You need to log in before you can comment on or make changes to this bug.