Bug 612204

Summary: Significant memory consumption by QMF python console applications.
Product: Red Hat Enterprise MRG Reporter: Ken Giusti <kgiusti>
Component: qpid-qmfAssignee: Ken Giusti <kgiusti>
Status: NEW --- QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: high Docs Contact:
Priority: low    
Version: betaCC: jross
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-03-10 18:29:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ken Giusti 2010-07-07 14:39:33 UTC 
Description of problem:

Under some circumstances, QMF console applications using the python console implementation consume huge amounts of memory.


Version-Release number of selected component (if applicable):
beta 1.3

How reproducible:
Not deterministice.

Steps to Reproduce:
1. Unknown
2.
3.
  
Actual results:

#
top - 09:45:40 up 184 days, 49 min,  1 user,  load average: 3.51, 3.87, 4.22
#
Tasks: 245 total,   1 running, 243 sleeping,   1 stopped,   0 zombie
#
Cpu(s):  0.3%us,  0.1%sy,  0.0%ni, 87.1%id, 12.4%wa,  0.0%hi,  0.1%si,  0.0%st
#
Mem:  12299640k total, 12234804k used,    64836k free,    39184k buffers
#
Swap: 18481144k total,  9270860k used,  9210284k free,   140668k cached
#
 
#
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
#
 6700 root      15   0 11.8g  10g 1940 S  0.7 93.2   1590:43 condor_configd     
#
 2007 qpidd     15   0 2048m 126m 2592 S  2.7  1.1  38:55.52 qpidd             
#
20860 postgres  15   0  118m 7572 7376 S  0.0  0.1   1:13.72 postmaster         


Expected results:
Memory usage is expected to be much lower.

Additional info:
Broker appears to be experiencing queue backups at the same time - potentially related?
Comment 1 Ken Giusti 2010-07-07 14:41:16 UTC 
pmap output of application exhibiting the problem:


   1.
      [root@mrg31 ~]# pmap -d 6700
   2.
      6700:   /usr/bin/python /usr/sbin/condor_configd
   3.
      Address           Kbytes Mode  Offset           Device    Mapping
   4.
      0000000000400000       4 r-x-- 0000000000000000 0fd:00000 python
   5.
      0000000000600000       4 rw--- 0000000000000000 0fd:00000 python
   6.
      000000000da17000  833348 rw--- 000000000da17000 000:00000   [ anon ]
   7.
      00000000407f0000       4 ----- 00000000407f0000 000:00000   [ anon ]
   8.
      00000000407f1000   10240 rw--- 00000000407f1000 000:00000   [ anon ]
   9.
      00000000411f1000       4 ----- 00000000411f1000 000:00000   [ anon ]
  10.
      00000000411f2000   10240 rw--- 00000000411f2000 000:00000   [ anon ]
  11.
      0000000041bf2000       4 ----- 0000000041bf2000 000:00000   [ anon ]
  12.
      0000000041bf3000   10240 rw--- 0000000041bf3000 000:00000   [ anon ]
  13.
      00000000425f3000       4 ----- 00000000425f3000 000:00000   [ anon ]
  14.
      00000000425f4000   10240 rw--- 00000000425f4000 000:00000   [ anon ]
  15.
      0000000042ff4000       4 ----- 0000000042ff4000 000:00000   [ anon ]
  16.
      0000000042ff5000   10240 rw--- 0000000042ff5000 000:00000   [ anon ]
  17.
      00000000439f5000       4 ----- 00000000439f5000 000:00000   [ anon ]
  18.
      00000000439f6000   10240 rw--- 00000000439f6000 000:00000   [ anon ]
  19.
      00000000443f6000       4 ----- 00000000443f6000 000:00000   [ anon ]
  20.
      00000000443f7000   10240 rw--- 00000000443f7000 000:00000   [ anon ]
  21.
      0000000044df7000       4 ----- 0000000044df7000 000:00000   [ anon ]
  22.
      0000000044df8000   10240 rw--- 0000000044df8000 000:00000   [ anon ]
  23.
      00000000457f8000       4 ----- 00000000457f8000 000:00000   [ anon ]
  24.
      00000000457f9000   10240 rw--- 00000000457f9000 000:00000   [ anon ]
  25.
      00000000461f9000       4 ----- 00000000461f9000 000:00000   [ anon ]
  26.
      00000000461fa000   10240 rw--- 00000000461fa000 000:00000   [ anon ]
  27.
      0000000046bfa000       4 ----- 0000000046bfa000 000:00000   [ anon ]
  28.
      0000000046bfb000   10240 rw--- 0000000046bfb000 000:00000   [ anon ]
  29.
      00000000475fb000       4 ----- 00000000475fb000 000:00000   [ anon ]
  30.
      00000000475fc000   10240 rw--- 00000000475fc000 000:00000   [ anon ]
  31.
      0000000047ffc000       4 ----- 0000000047ffc000 000:00000   [ anon ]
  32.
      0000000047ffd000   10240 rw--- 0000000047ffd000 000:00000   [ anon ]
  33.
      00000038aec00000     112 r-x-- 0000000000000000 0fd:00000 ld-2.5.so
  34.
      00000038aee1b000       4 r---- 000000000001b000 0fd:00000 ld-2.5.so
  35.
      00000038aee1c000       4 rw--- 000000000001c000 0fd:00000 ld-2.5.so
  36.
      00000038af000000    1332 r-x-- 0000000000000000 0fd:00000 libc-2.5.so
  37.
      00000038af14d000    2048 ----- 000000000014d000 0fd:00000 libc-2.5.so
  38.
      00000038af34d000      16 r---- 000000000014d000 0fd:00000 libc-2.5.so
  39.
      00000038af351000       4 rw--- 0000000000151000 0fd:00000 libc-2.5.so
  40.
      00000038af352000      20 rw--- 00000038af352000 000:00000   [ anon ]
  41.
      00000038af400000     520 r-x-- 0000000000000000 0fd:00000 libm-2.5.so
  42.
      00000038af482000    2044 ----- 0000000000082000 0fd:00000 libm-2.5.so
  43.
      00000038af681000       4 r---- 0000000000081000 0fd:00000 libm-2.5.so
  44.
      00000038af682000       4 rw--- 0000000000082000 0fd:00000 libm-2.5.so
  45.
      00000038af800000       8 r-x-- 0000000000000000 0fd:00000 libdl-2.5.so
  46.
      00000038af802000    2048 ----- 0000000000002000 0fd:00000 libdl-2.5.so
  47.
      00000038afa02000       4 r---- 0000000000002000 0fd:00000 libdl-2.5.so
  48.
      00000038afa03000       4 rw--- 0000000000003000 0fd:00000 libdl-2.5.so
  49.
      00000038afc00000      88 r-x-- 0000000000000000 0fd:00000 libpthread-2.5.so
  50.
      00000038afc16000    2044 ----- 0000000000016000 0fd:00000 libpthread-2.5.so
  51.
      00000038afe15000       4 r---- 0000000000015000 0fd:00000 libpthread-2.5.so
  52.
      00000038afe16000       4 rw--- 0000000000016000 0fd:00000 libpthread-2.5.so
  53.
      00000038afe17000      16 rw--- 00000038afe17000 000:00000   [ anon ]
  54.
      00000038b0000000      80 r-x-- 0000000000000000 0fd:00000 libz.so.1.2.3
  55.
      00000038b0014000    2044 ----- 0000000000014000 0fd:00000 libz.so.1.2.3
  56.
      00000038b0213000       4 rw--- 0000000000013000 0fd:00000 libz.so.1.2.3
  57.
      00000038b0800000     236 r-x-- 0000000000000000 0fd:00000 libsepol.so.1
  58.
      00000038b083b000    2048 ----- 000000000003b000 0fd:00000 libsepol.so.1
  59.
      00000038b0a3b000       4 rw--- 000000000003b000 0fd:00000 libsepol.so.1
  60.
      00000038b0a3c000      40 rw--- 00000038b0a3c000 000:00000   [ anon ]
  61.
      00000038b0c00000      84 r-x-- 0000000000000000 0fd:00000 libselinux.so.1
  62.
      00000038b0c15000    2048 ----- 0000000000015000 0fd:00000 libselinux.so.1
  63.
      00000038b0e15000       8 rw--- 0000000000015000 0fd:00000 libselinux.so.1
  64.
      00000038b0e17000       4 rw--- 00000038b0e17000 000:00000   [ anon ]
  65.
      00000038b3400000    1204 r-x-- 0000000000000000 0fd:00000 libcrypto.so.0.9.8e
  66.
      00000038b352d000    2044 ----- 000000000012d000 0fd:00000 libcrypto.so.0.9.8e
  67.
      00000038b372c000     132 rw--- 000000000012c000 0fd:00000 libcrypto.so.0.9.8e
  68.
      00000038b374d000      16 rw--- 00000038b374d000 000:00000   [ anon ]
  69.
      00000038b3c00000       8 r-x-- 0000000000000000 0fd:00000 libcom_err.so.2.1
  70.
      00000038b3c02000    2044 ----- 0000000000002000 0fd:00000 libcom_err.so.2.1
  71.
      00000038b3e01000       4 rw--- 0000000000001000 0fd:00000 libcom_err.so.2.1
  72.
      00000038b4000000      68 r-x-- 0000000000000000 0fd:00000 libresolv-2.5.so
  73.
      00000038b4011000    2048 ----- 0000000000011000 0fd:00000 libresolv-2.5.so
  74.
      00000038b4211000       4 r---- 0000000000011000 0fd:00000 libresolv-2.5.so
  75.
      00000038b4212000       4 rw--- 0000000000012000 0fd:00000 libresolv-2.5.so
  76.
      00000038b4213000       8 rw--- 00000038b4213000 000:00000   [ anon ]
  77.
      00000038b6400000       8 r-x-- 0000000000000000 0fd:00000 libkeyutils-1.2.so
  78.
      00000038b6402000    2044 ----- 0000000000002000 0fd:00000 libkeyutils-1.2.so
  79.
      00000038b6601000       4 rw--- 0000000000001000 0fd:00000 libkeyutils-1.2.so
  80.
      00000038b6800000      32 r-x-- 0000000000000000 0fd:00000 libkrb5support.so.0.1
  81.
      00000038b6808000    2044 ----- 0000000000008000 0fd:00000 libkrb5support.so.0.1
  82.
      00000038b6a07000       4 rw--- 0000000000007000 0fd:00000 libkrb5support.so.0.1
  83.
      00000038b6c00000     580 r-x-- 0000000000000000 0fd:00000 libkrb5.so.3.3
  84.
      00000038b6c91000    2048 ----- 0000000000091000 0fd:00000 libkrb5.so.3.3
  85.
      00000038b6e91000      16 rw--- 0000000000091000 0fd:00000 libkrb5.so.3.3
  86.
      00000038b7400000     276 r-x-- 0000000000000000 0fd:00000 libssl.so.0.9.8e
  87.
      00000038b7445000    2044 ----- 0000000000045000 0fd:00000 libssl.so.0.9.8e
  88.
      00000038b7644000      24 rw--- 0000000000044000 0fd:00000 libssl.so.0.9.8e
  89.
      00000038b7800000     176 r-x-- 0000000000000000 0fd:00000 libgssapi_krb5.so.2.2
  90.
      00000038b782c000    2048 ----- 000000000002c000 0fd:00000 libgssapi_krb5.so.2.2
  91.
      00000038b7a2c000       8 rw--- 000000000002c000 0fd:00000 libgssapi_krb5.so.2.2
  92.
      00000038b8e00000     144 r-x-- 0000000000000000 0fd:00000 libk5crypto.so.3.1
  93.
      00000038b8e24000    2044 ----- 0000000000024000 0fd:00000 libk5crypto.so.3.1
  94.
      00000038b9023000       8 rw--- 0000000000023000 0fd:00000 libk5crypto.so.3.1
  95.
      00000038bce00000       8 r-x-- 0000000000000000 0fd:00000 libutil-2.5.so
  96.
      00000038bce02000    2044 ----- 0000000000002000 0fd:00000 libutil-2.5.so
  97.
      00000038bd001000       4 r---- 0000000000001000 0fd:00000 libutil-2.5.so
  98.
      00000038bd002000       4 rw--- 0000000000002000 0fd:00000 libutil-2.5.so
  99.
      00000038be200000    1024 r-x-- 0000000000000000 0fd:00000 libpython2.4.so.1.0
 100.
      00000038be300000    2048 ----- 0000000000100000 0fd:00000 libpython2.4.so.1.0
 101.
      00000038be500000     180 rw--- 0000000000100000 0fd:00000 libpython2.4.so.1.0
 102.
      00000038be52d000      20 rw--- 00000038be52d000 000:00000   [ anon ]
 103.
      00002aaaaaaab000   21840 rw--- 00002aaaaaaab000 000:00000   [ anon ]
 104.
      00002aaaac000000     132 rw--- 00002aaaac000000 000:00000   [ anon ]
 105.
      00002aaaac021000   65404 ----- 00002aaaac021000 000:00000   [ anon ]
 106.
      00002aaab0000000 2880960 rw--- 00002aaab0000000 000:00000   [ anon ]
 107.
      00002aab5fd71000 2103728 rw--- 00002aab5fd71000 000:00000   [ anon ]
 108.
      00002aabe03de000  916436 rw--- 00002aabe03de000 000:00000   [ anon ]
 109.
      00002aac18311000 3342620 rw--- 00002aac18311000 000:00000   [ anon ]
 110.
      00002aace4396000  971244 rw--- 00002aace4396000 000:00000   [ anon ]
 111.
      00002aad1f812000  262856 rw--- 00002aad1f812000 000:00000   [ anon ]
 112.
      00002aad2f943000  676568 rw--- 00002aad2f943000 000:00000   [ anon ]
 113.
      00002aad58dfa000   65584 rw--- 00002aad58dfa000 000:00000   [ anon ]
 114.
      00002af42ab49000       4 rw--- 00002af42ab49000 000:00000   [ anon ]
 115.
      00002af42ab5d000     276 rw--- 00002af42ab5d000 000:00000   [ anon ]
 116.
      00002af42aba3000     260 rw--- 00002af42aba3000 000:00000   [ anon ]
 117.
      00002af42ac16000   55144 r---- 0000000000000000 0fd:00000 locale-archive
 118.
      00002af42e1f1000     520 rw--- 00002af42e1f1000 000:00000   [ anon ]
 119.
      00002af42e273000      12 r-x-- 0000000000000000 0fd:00000 timemodule.so
 120.
      00002af42e276000    2048 ----- 0000000000003000 0fd:00000 timemodule.so
 121.
      00002af42e476000       8 rw--- 0000000000003000 0fd:00000 timemodule.so
 122.
      00002af42e478000      20 r-x-- 0000000000000000 0fd:00000 stropmodule.so
 123.
      00002af42e47d000    2044 ----- 0000000000005000 0fd:00000 stropmodule.so
 124.
      00002af42e67c000       8 rw--- 0000000000004000 0fd:00000 stropmodule.so
 125.
      00002af42e67e000      12 r-x-- 0000000000000000 0fd:00000 cStringIO.so
 126.
      00002af42e681000    2048 ----- 0000000000003000 0fd:00000 cStringIO.so
 127.
      00002af42e881000       8 rw--- 0000000000003000 0fd:00000 cStringIO.so
 128.
      00002af42e883000      16 r-x-- 0000000000000000 0fd:00000 collectionsmodule.so
 129.
      00002af42e887000    2048 ----- 0000000000004000 0fd:00000 collectionsmodule.so
 130.
      00002af42ea87000       4 rw--- 0000000000004000 0fd:00000 collectionsmodule.so
 131.
      00002af42ea88000      40 r-x-- 0000000000000000 0fd:00000 _socketmodule.so
 132.
      00002af42ea92000    2044 ----- 000000000000a000 0fd:00000 _socketmodule.so
 133.
      00002af42ec91000      16 rw--- 0000000000009000 0fd:00000 _socketmodule.so
 134.
      00002af42ec95000      16 r-x-- 0000000000000000 0fd:00000 _ssl.so
 135.
      00002af42ec99000    2044 ----- 0000000000004000 0fd:00000 _ssl.so
 136.
      00002af42ee98000       4 rw--- 0000000000003000 0fd:00000 _ssl.so
 137.
      00002af42ee99000      12 r-x-- 0000000000000000 0fd:00000 mathmodule.so
 138.
      00002af42ee9c000    2048 ----- 0000000000003000 0fd:00000 mathmodule.so
 139.
      00002af42f09c000       4 rw--- 0000000000003000 0fd:00000 mathmodule.so
 140.
      00002af42f09d000      20 r-x-- 0000000000000000 0fd:00000 binascii.so
 141.
      00002af42f0a2000    2044 ----- 0000000000005000 0fd:00000 binascii.so
 142.
      00002af42f2a1000       4 rw--- 0000000000004000 0fd:00000 binascii.so
 143.
      00002af42f2a2000      12 r-x-- 0000000000000000 0fd:00000 _randommodule.so
 144.
      00002af42f2a5000    2044 ----- 0000000000003000 0fd:00000 _randommodule.so
 145.
      00002af42f4a4000       4 rw--- 0000000000002000 0fd:00000 _randommodule.so
 146.
      00002af42f4a5000      12 r-x-- 0000000000000000 0fd:00000 fcntlmodule.so
 147.
      00002af42f4a8000    2044 ----- 0000000000003000 0fd:00000 fcntlmodule.so
 148.
      00002af42f6a7000       8 rw--- 0000000000002000 0fd:00000 fcntlmodule.so
 149.
      00002af42f6aa000     260 rw--- 00002af42f6aa000 000:00000   [ anon ]
 150.
      00002af42f6eb000      20 r-x-- 0000000000000000 0fd:00000 structmodule.so
 151.
      00002af42f6f0000    2044 ----- 0000000000005000 0fd:00000 structmodule.so
 152.
      00002af42f8ef000       8 rw--- 0000000000004000 0fd:00000 structmodule.so
 153.
      00002af42f8f1000      60 r-x-- 0000000000000000 0fd:00000 datetime.so
 154.
      00002af42f900000    2044 ----- 000000000000f000 0fd:00000 datetime.so
 155.
      00002af42faff000      16 rw--- 000000000000e000 0fd:00000 datetime.so
 156.
      00002af42fb03000     260 rw--- 00002af42fb03000 000:00000   [ anon ]
 157.
      00002af42fb44000      68 r-x-- 0000000000000000 0fd:00000 cPickle.so
 158.
      00002af42fb55000    2048 ----- 0000000000011000 0fd:00000 cPickle.so
 159.
      00002af42fd55000       4 rw--- 0000000000011000 0fd:00000 cPickle.so
 160.
      00002af42fd57000     520 rw--- 00002af42fd57000 000:00000   [ anon ]
 161.
      00002af42fdda000    1032 rw--- 00002af42fdda000 000:00000   [ anon ]
 162.
      00002af42fedc000       8 r-x-- 0000000000000000 0fd:00000 grpmodule.so
 163.
      00002af42fede000    2044 ----- 0000000000002000 0fd:00000 grpmodule.so
 164.
      00002af4300dd000       4 rw--- 0000000000001000 0fd:00000 grpmodule.so
 165.
      00002af4300de000      16 r-x-- 0000000000000000 0fd:00000 zlibmodule.so
 166.
      00002af4300e2000    2044 ----- 0000000000004000 0fd:00000 zlibmodule.so
 167.
      00002af4302e1000       8 rw--- 0000000000003000 0fd:00000 zlibmodule.so
 168.
      00002af4302e3000      12 r-x-- 0000000000000000 0fd:00000 selectmodule.so
 169.
      00002af4302e6000    2048 ----- 0000000000003000 0fd:00000 selectmodule.so
 170.
      00002af4304e6000       4 rw--- 0000000000003000 0fd:00000 selectmodule.so
 171.
      00002af4304e8000     264 rw--- 00002af4304e8000 000:00000   [ anon ]
 172.
      00002af43053d000      40 r-x-- 0000000000000000 0fd:00000 libnss_files-2.5.so
 173.
      00002af430547000    2044 ----- 000000000000a000 0fd:00000 libnss_files-2.5.so
 174.
      00002af430746000       4 r---- 0000000000009000 0fd:00000 libnss_files-2.5.so
 175.
      00002af430747000       4 rw--- 000000000000a000 0fd:00000 libnss_files-2.5.so
 176.
      00002af430748000      16 r-x-- 0000000000000000 0fd:00000 libnss_dns-2.5.so
 177.
      00002af43074c000    2044 ----- 0000000000004000 0fd:00000 libnss_dns-2.5.so
 178.
      00002af43094b000       4 r---- 0000000000003000 0fd:00000 libnss_dns-2.5.so
 179.
      00002af43094c000       4 rw--- 0000000000004000 0fd:00000 libnss_dns-2.5.so
 180.
      00007fff2f2a0000      84 rw--- 00007ffffffea000 000:00000   [ stack ]
 181.
      ffffffffff600000    8192 ----- 0000000000000000 000:00000   [ anon ]
 182.
      mapped: 12421472K    writeable/private: 12212588K    shared: 0K
Comment 2 Matthew Farrellee 2010-07-07 18:23:07 UTC 
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 6700 root      15   0 11.8g  10g 1940 S  0.7 93.2   1590:43 condor_configd     

I'd note that the condor_configd typically uses a vanishingly small amount of CPU. Some abnormal event must have occurred for it to accumulate so much TIME.