Bug 1359506 - Small memory leak when trying to over-subscribe queue with exclusive subscription
Summary: Small memory leak when trying to over-subscribe queue with exclusive subscrip...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: 2.5
Hardware: x86_64
OS: Linux
high
high
Target Milestone: 3.2.2
: ---
Assignee: Cliff Jansen
QA Contact: Messaging QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-24 12:41 UTC by Pavel Moravec
Modified: 2016-10-11 07:36 UTC (History)
5 users (show)

Fixed In Version: qpid-cpp-0.34-17
Doc Type: Bug Fix
Doc Text:
Cause: If any worker thread is idle for long periods of time, certain routine memory flushing may not happen, making it look like a memory leak. Consequence: memory is not freed when it should be Fix: The code has been fixed so that all worker threads are made active at least once per minute Result: memory is released regularly and does not accumulate unnecessarily
Clone Of:
Environment:
Last Closed: 2016-10-11 07:36:57 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2049 normal SHIPPED_LIVE Red Hat Enterprise MRG Messaging 3.2.2 Bug Fix Release 2016-10-11 11:36:01 UTC

Description Pavel Moravec 2016-07-24 12:41:30 UTC
Description of problem:
When (exclusively) subscribing to a (nonexclusive) queue with another exclusive consumer, a small memory leak occurs.


Version-Release number of selected component (if applicable):
qpid-cpp-server-0.18-39.el6.x86_64


How reproducible:
100%


Steps to Reproduce:
1. Create exclusive subscriber:
qpid-receive -a "q; { link: { x-subscribe : { exclusive:true } }, create:always}" -f &

2. Attempt to subscribe to the same queue in a loop:
while true; do qpid-receive -a "q; { link: { x-subscribe : { exclusive:true } }, create:always}" >/dev/null 2>&1; sleep 0.1; done

3. in another terminal, monitor memory:
while true; do ps aux | grep qpidd | grep daemon | awk -F "--config" '{ print $1 }'; sleep 60; done


Actual results:
3. shows small but constant memory usage growth


Expected results:
3. to show no memory growth


Additional info:

Comment 1 Pavel Moravec 2016-07-24 13:50:21 UTC
The leak is present also in qpid-cpp-server-0.34-15.el6.x86_64 but it is not present in current qpid upstream (sic!).

Comment 2 Pavel Moravec 2016-07-25 07:19:48 UTC
Long test showed the leak can be present also in upstream - ps shows little bit confusing data..

ps output (just RSS column) taken every minute from a test against upstream broker:

4168
8120
8164
8416
8444
8568
8576
8672
8708
8816
8852
8892
8916
8988
9040
9192
9272
9308
9484
9576
9636
9680
9812
9868
9940
9984
9980
10116
10164
10220
10304
10356
10428
10516
10528
10576
10648
10668
10732
10784
10844
10936
10992
11040
11188
11264
11324
11384
11456
11508
11540
11592
11640
11676
11788
11820
11884
11920
12000
12100
12152
12176
12272
12304
12364
12388
12460
12508
12556
12636
12692
12724
12772
12816
12896
12960
13044
13080
13164
13276
13392
13392
13400
13444
13452
13460
13416
13540
13548
13572
13568
13588
13588
13624
13660
13696
13696
13704
13724
13728
13724
13744
13772
13780
13800
13848
13844
13844
13848
13864
13912
13924
13932
14168
14176
14228
14232
14252
14256
14304
14332
14340
14376
14392
14444
14456
14480
14508
14528
14544
14568
14584
14612
14640
14648
14672
14712
14720
14740
14784
14800
14800
14840
14876
14912
14964
14972
14988
15140
15152
15196
15216
15240
15292
15356
15436
15448
15544
15576
15624
15664
15688
15740
15768
15832
15896
15904
15944
15964
15988
16004
16028
16068
16120
16056
16056
16060
16076
16088
16084
16088
16136
16136
16172
16172
16172
16172
16168
16172
16176
16184
16180
16180
16180
16180
16176
16176
16176
16176
16176
16172
16172
16200
16200
16200
16196
16196
16196
16196
16192
16192
16192
16192
16192
16192
16192
16188
16188
16212
16208
16232
16240
16244
16232
16288
16300
16300
16296
16296
16296
16296
16308
16308
16308
16324
16368
16372
16432
16436
16436
16448
16476
16476
16500
17860
17896
17896
17896
17900
17896
17900
17900
17916
17920
17932
17948
17260
17016
17012
17012
17048
17012
17012
17012
17008
17008
17008
17008
17008
17008
17004
17004
17004
17000
17000
17000
17000
17000
17000
17000
17004
17004
17004
17004
17004
17004
17000
16224
16224
16220
16220
16220
16216
16216
16216
16216
16212
16212
16212
16208
16208
16208
16208
16204
16204
16204
16204
16204
16204
16204
16200
16200
16200
16200
16196
16196
16244
16244
16240
16240
16240
16236
16372
16380
16384
16392
16400
16404
16408
16416
16348
16348
16348
16348
16344
16344
16344
16344
16340
16340
16340
16340
16340
16336
16336
16336
16336
16336
16336
16332
16332
16332
16332
16332
16332
16332
16332
16332
16328
16328
16328
16328
16328
16324
16324
16320
16368
16368
16368
16368
16364
16364
16364
16364
16360
16340
16340
16340
16340
16336
16336
16336
16336
16336
16332
16332

Comment 3 Gordon Sim 2016-07-26 15:43:15 UTC
(In reply to Pavel Moravec from comment #2)
> Long test showed the leak can be present also in upstream - ps shows little
> bit confusing data..
> 
> ps output (just RSS column) taken every minute from a test against upstream
> broker:
> 
> 4168
> 8120
> 8164
> 8416
> 8444
> 8568
> 8576
> 8672
> 8708
> 8816
> 8852
> 8892
> 8916
> 8988
> 9040
> 9192
> 9272
> 9308
> 9484
> 9576
> 9636
> 9680
> 9812
> 9868
> 9940
> 9984
> 9980
> 10116
> 10164
> 10220
> 10304
> 10356
> 10428
> 10516
> 10528
> 10576
> 10648
> 10668
> 10732
> 10784
> 10844
> 10936
> 10992
> 11040
> 11188
> 11264
> 11324
> 11384
> 11456
> 11508
> 11540
> 11592
> 11640
> 11676
> 11788
> 11820
> 11884
> 11920
> 12000
> 12100
> 12152
> 12176
> 12272
> 12304
> 12364
> 12388
> 12460
> 12508
> 12556
> 12636
> 12692
> 12724
> 12772
> 12816
> 12896
> 12960
> 13044
> 13080
> 13164
> 13276
> 13392
> 13392
> 13400
> 13444
> 13452
> 13460
> 13416
> 13540
> 13548
> 13572
> 13568
> 13588
> 13588
> 13624
> 13660
> 13696
> 13696
> 13704
> 13724
> 13728
> 13724
> 13744
> 13772
> 13780
> 13800
> 13848
> 13844
> 13844
> 13848
> 13864
> 13912
> 13924
> 13932
> 14168
> 14176
> 14228
> 14232
> 14252
> 14256
> 14304
> 14332
> 14340
> 14376
> 14392
> 14444
> 14456
> 14480
> 14508
> 14528
> 14544
> 14568
> 14584
> 14612
> 14640
> 14648
> 14672
> 14712
> 14720
> 14740
> 14784
> 14800
> 14800
> 14840
> 14876
> 14912
> 14964
> 14972
> 14988
> 15140
> 15152
> 15196
> 15216
> 15240
> 15292
> 15356
> 15436
> 15448
> 15544
> 15576
> 15624
> 15664
> 15688
> 15740
> 15768
> 15832
> 15896
> 15904
> 15944
> 15964
> 15988
> 16004
> 16028
> 16068
> 16120
> 16056
> 16056
> 16060
> 16076
> 16088
> 16084
> 16088
> 16136
> 16136
> 16172
> 16172
> 16172
> 16172
> 16168
> 16172
> 16176
> 16184
> 16180
> 16180
> 16180
> 16180
> 16176
> 16176
> 16176
> 16176
> 16176
> 16172
> 16172
> 16200
> 16200
> 16200
> 16196
> 16196
> 16196
> 16196
> 16192
> 16192
> 16192
> 16192
> 16192
> 16192
> 16192
> 16188
> 16188
> 16212
> 16208
> 16232
> 16240
> 16244
> 16232
> 16288
> 16300
> 16300
> 16296
> 16296
> 16296
> 16296
> 16308
> 16308
> 16308
> 16324
> 16368
> 16372
> 16432
> 16436
> 16436
> 16448
> 16476
> 16476
> 16500
> 17860
> 17896
> 17896
> 17896
> 17900
> 17896
> 17900
> 17900
> 17916
> 17920
> 17932
> 17948
> 17260
> 17016
> 17012
> 17012
> 17048
> 17012
> 17012
> 17012
> 17008
> 17008
> 17008
> 17008
> 17008
> 17008
> 17004
> 17004
> 17004
> 17000
> 17000
> 17000
> 17000
> 17000
> 17000
> 17000
> 17004
> 17004
> 17004
> 17004
> 17004
> 17004
> 17000
> 16224
> 16224
> 16220
> 16220
> 16220
> 16216
> 16216
> 16216
> 16216
> 16212
> 16212
> 16212
> 16208
> 16208
> 16208
> 16208
> 16204
> 16204
> 16204
> 16204
> 16204
> 16204
> 16204
> 16200
> 16200
> 16200
> 16200
> 16196
> 16196
> 16244
> 16244
> 16240
> 16240
> 16240
> 16236
> 16372
> 16380
> 16384
> 16392
> 16400
> 16404
> 16408
> 16416
> 16348
> 16348
> 16348
> 16348
> 16344
> 16344
> 16344
> 16344
> 16340
> 16340
> 16340
> 16340
> 16340
> 16336
> 16336
> 16336
> 16336
> 16336
> 16336
> 16332
> 16332
> 16332
> 16332
> 16332
> 16332
> 16332
> 16332
> 16332
> 16328
> 16328
> 16328
> 16328
> 16328
> 16324
> 16324
> 16320
> 16368
> 16368
> 16368
> 16368
> 16364
> 16364
> 16364
> 16364
> 16360
> 16340
> 16340
> 16340
> 16340
> 16336
> 16336
> 16336
> 16336
> 16336
> 16332
> 16332

I'm not sure this is actually a leak. The last 200 values recorded for memory (i.e. 3 hours and 20 minutes worth) show a very small difference and the memory goes down as well as up.

I see similar numbers just opening and closing a receiver - i.e. with no failed attempt at exclusive queue subscription. I think the initial growth may just be memory pressure from creating and deleting the various objects for connection, session & susbscription very rapidly. Once the available heap has expanded sufficiently, extra memory is not required.

This may of course be different from the originally reported issue on -18 or -34.

Comment 9 Cliff Jansen 2016-08-02 07:30:07 UTC
https://issues.apache.org/jira/browse/QPID-7373 may be the same issue.  Note that that bug depends solely on creating many sequential connections and is unrelated to exclusive access to a queue.

Since my proof of concept fix for QPID-7373 seems to address the memory leak in the provided reproducer in my environment (RHEL6, 6core/12thread cpu), I am hopeful the problems may be the same.

Comment 13 Pavel Moravec 2016-08-05 16:04:35 UTC
Recommendation for QE: it is worth testing it with different number of worker threads. I.e. once with default number, then with 1 and also with say 10.

(preliminary results of mine testing are the leak is fixed)

Comment 16 errata-xmlrpc 2016-10-11 07:36:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2049.html


Note You need to log in before you can comment on or make changes to this bug.