Description of problem: Under some circumstances, QMF console applications using the python console implementation consume huge amounts of memory. Version-Release number of selected component (if applicable): beta 1.3 How reproducible: Not deterministice. Steps to Reproduce: 1. Unknown 2. 3. Actual results: # top - 09:45:40 up 184 days, 49 min, 1 user, load average: 3.51, 3.87, 4.22 # Tasks: 245 total, 1 running, 243 sleeping, 1 stopped, 0 zombie # Cpu(s): 0.3%us, 0.1%sy, 0.0%ni, 87.1%id, 12.4%wa, 0.0%hi, 0.1%si, 0.0%st # Mem: 12299640k total, 12234804k used, 64836k free, 39184k buffers # Swap: 18481144k total, 9270860k used, 9210284k free, 140668k cached # # PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND # 6700 root 15 0 11.8g 10g 1940 S 0.7 93.2 1590:43 condor_configd # 2007 qpidd 15 0 2048m 126m 2592 S 2.7 1.1 38:55.52 qpidd # 20860 postgres 15 0 118m 7572 7376 S 0.0 0.1 1:13.72 postmaster Expected results: Memory usage is expected to be much lower. Additional info: Broker appears to be experiencing queue backups at the same time - potentially related?
pmap output of application exhibiting the problem: 1. [root@mrg31 ~]# pmap -d 6700 2. 6700: /usr/bin/python /usr/sbin/condor_configd 3. Address Kbytes Mode Offset Device Mapping 4. 0000000000400000 4 r-x-- 0000000000000000 0fd:00000 python 5. 0000000000600000 4 rw--- 0000000000000000 0fd:00000 python 6. 000000000da17000 833348 rw--- 000000000da17000 000:00000 [ anon ] 7. 00000000407f0000 4 ----- 00000000407f0000 000:00000 [ anon ] 8. 00000000407f1000 10240 rw--- 00000000407f1000 000:00000 [ anon ] 9. 00000000411f1000 4 ----- 00000000411f1000 000:00000 [ anon ] 10. 00000000411f2000 10240 rw--- 00000000411f2000 000:00000 [ anon ] 11. 0000000041bf2000 4 ----- 0000000041bf2000 000:00000 [ anon ] 12. 0000000041bf3000 10240 rw--- 0000000041bf3000 000:00000 [ anon ] 13. 00000000425f3000 4 ----- 00000000425f3000 000:00000 [ anon ] 14. 00000000425f4000 10240 rw--- 00000000425f4000 000:00000 [ anon ] 15. 0000000042ff4000 4 ----- 0000000042ff4000 000:00000 [ anon ] 16. 0000000042ff5000 10240 rw--- 0000000042ff5000 000:00000 [ anon ] 17. 00000000439f5000 4 ----- 00000000439f5000 000:00000 [ anon ] 18. 00000000439f6000 10240 rw--- 00000000439f6000 000:00000 [ anon ] 19. 00000000443f6000 4 ----- 00000000443f6000 000:00000 [ anon ] 20. 00000000443f7000 10240 rw--- 00000000443f7000 000:00000 [ anon ] 21. 0000000044df7000 4 ----- 0000000044df7000 000:00000 [ anon ] 22. 0000000044df8000 10240 rw--- 0000000044df8000 000:00000 [ anon ] 23. 00000000457f8000 4 ----- 00000000457f8000 000:00000 [ anon ] 24. 00000000457f9000 10240 rw--- 00000000457f9000 000:00000 [ anon ] 25. 00000000461f9000 4 ----- 00000000461f9000 000:00000 [ anon ] 26. 00000000461fa000 10240 rw--- 00000000461fa000 000:00000 [ anon ] 27. 0000000046bfa000 4 ----- 0000000046bfa000 000:00000 [ anon ] 28. 0000000046bfb000 10240 rw--- 0000000046bfb000 000:00000 [ anon ] 29. 00000000475fb000 4 ----- 00000000475fb000 000:00000 [ anon ] 30. 00000000475fc000 10240 rw--- 00000000475fc000 000:00000 [ anon ] 31. 0000000047ffc000 4 ----- 0000000047ffc000 000:00000 [ anon ] 32. 0000000047ffd000 10240 rw--- 0000000047ffd000 000:00000 [ anon ] 33. 00000038aec00000 112 r-x-- 0000000000000000 0fd:00000 ld-2.5.so 34. 00000038aee1b000 4 r---- 000000000001b000 0fd:00000 ld-2.5.so 35. 00000038aee1c000 4 rw--- 000000000001c000 0fd:00000 ld-2.5.so 36. 00000038af000000 1332 r-x-- 0000000000000000 0fd:00000 libc-2.5.so 37. 00000038af14d000 2048 ----- 000000000014d000 0fd:00000 libc-2.5.so 38. 00000038af34d000 16 r---- 000000000014d000 0fd:00000 libc-2.5.so 39. 00000038af351000 4 rw--- 0000000000151000 0fd:00000 libc-2.5.so 40. 00000038af352000 20 rw--- 00000038af352000 000:00000 [ anon ] 41. 00000038af400000 520 r-x-- 0000000000000000 0fd:00000 libm-2.5.so 42. 00000038af482000 2044 ----- 0000000000082000 0fd:00000 libm-2.5.so 43. 00000038af681000 4 r---- 0000000000081000 0fd:00000 libm-2.5.so 44. 00000038af682000 4 rw--- 0000000000082000 0fd:00000 libm-2.5.so 45. 00000038af800000 8 r-x-- 0000000000000000 0fd:00000 libdl-2.5.so 46. 00000038af802000 2048 ----- 0000000000002000 0fd:00000 libdl-2.5.so 47. 00000038afa02000 4 r---- 0000000000002000 0fd:00000 libdl-2.5.so 48. 00000038afa03000 4 rw--- 0000000000003000 0fd:00000 libdl-2.5.so 49. 00000038afc00000 88 r-x-- 0000000000000000 0fd:00000 libpthread-2.5.so 50. 00000038afc16000 2044 ----- 0000000000016000 0fd:00000 libpthread-2.5.so 51. 00000038afe15000 4 r---- 0000000000015000 0fd:00000 libpthread-2.5.so 52. 00000038afe16000 4 rw--- 0000000000016000 0fd:00000 libpthread-2.5.so 53. 00000038afe17000 16 rw--- 00000038afe17000 000:00000 [ anon ] 54. 00000038b0000000 80 r-x-- 0000000000000000 0fd:00000 libz.so.1.2.3 55. 00000038b0014000 2044 ----- 0000000000014000 0fd:00000 libz.so.1.2.3 56. 00000038b0213000 4 rw--- 0000000000013000 0fd:00000 libz.so.1.2.3 57. 00000038b0800000 236 r-x-- 0000000000000000 0fd:00000 libsepol.so.1 58. 00000038b083b000 2048 ----- 000000000003b000 0fd:00000 libsepol.so.1 59. 00000038b0a3b000 4 rw--- 000000000003b000 0fd:00000 libsepol.so.1 60. 00000038b0a3c000 40 rw--- 00000038b0a3c000 000:00000 [ anon ] 61. 00000038b0c00000 84 r-x-- 0000000000000000 0fd:00000 libselinux.so.1 62. 00000038b0c15000 2048 ----- 0000000000015000 0fd:00000 libselinux.so.1 63. 00000038b0e15000 8 rw--- 0000000000015000 0fd:00000 libselinux.so.1 64. 00000038b0e17000 4 rw--- 00000038b0e17000 000:00000 [ anon ] 65. 00000038b3400000 1204 r-x-- 0000000000000000 0fd:00000 libcrypto.so.0.9.8e 66. 00000038b352d000 2044 ----- 000000000012d000 0fd:00000 libcrypto.so.0.9.8e 67. 00000038b372c000 132 rw--- 000000000012c000 0fd:00000 libcrypto.so.0.9.8e 68. 00000038b374d000 16 rw--- 00000038b374d000 000:00000 [ anon ] 69. 00000038b3c00000 8 r-x-- 0000000000000000 0fd:00000 libcom_err.so.2.1 70. 00000038b3c02000 2044 ----- 0000000000002000 0fd:00000 libcom_err.so.2.1 71. 00000038b3e01000 4 rw--- 0000000000001000 0fd:00000 libcom_err.so.2.1 72. 00000038b4000000 68 r-x-- 0000000000000000 0fd:00000 libresolv-2.5.so 73. 00000038b4011000 2048 ----- 0000000000011000 0fd:00000 libresolv-2.5.so 74. 00000038b4211000 4 r---- 0000000000011000 0fd:00000 libresolv-2.5.so 75. 00000038b4212000 4 rw--- 0000000000012000 0fd:00000 libresolv-2.5.so 76. 00000038b4213000 8 rw--- 00000038b4213000 000:00000 [ anon ] 77. 00000038b6400000 8 r-x-- 0000000000000000 0fd:00000 libkeyutils-1.2.so 78. 00000038b6402000 2044 ----- 0000000000002000 0fd:00000 libkeyutils-1.2.so 79. 00000038b6601000 4 rw--- 0000000000001000 0fd:00000 libkeyutils-1.2.so 80. 00000038b6800000 32 r-x-- 0000000000000000 0fd:00000 libkrb5support.so.0.1 81. 00000038b6808000 2044 ----- 0000000000008000 0fd:00000 libkrb5support.so.0.1 82. 00000038b6a07000 4 rw--- 0000000000007000 0fd:00000 libkrb5support.so.0.1 83. 00000038b6c00000 580 r-x-- 0000000000000000 0fd:00000 libkrb5.so.3.3 84. 00000038b6c91000 2048 ----- 0000000000091000 0fd:00000 libkrb5.so.3.3 85. 00000038b6e91000 16 rw--- 0000000000091000 0fd:00000 libkrb5.so.3.3 86. 00000038b7400000 276 r-x-- 0000000000000000 0fd:00000 libssl.so.0.9.8e 87. 00000038b7445000 2044 ----- 0000000000045000 0fd:00000 libssl.so.0.9.8e 88. 00000038b7644000 24 rw--- 0000000000044000 0fd:00000 libssl.so.0.9.8e 89. 00000038b7800000 176 r-x-- 0000000000000000 0fd:00000 libgssapi_krb5.so.2.2 90. 00000038b782c000 2048 ----- 000000000002c000 0fd:00000 libgssapi_krb5.so.2.2 91. 00000038b7a2c000 8 rw--- 000000000002c000 0fd:00000 libgssapi_krb5.so.2.2 92. 00000038b8e00000 144 r-x-- 0000000000000000 0fd:00000 libk5crypto.so.3.1 93. 00000038b8e24000 2044 ----- 0000000000024000 0fd:00000 libk5crypto.so.3.1 94. 00000038b9023000 8 rw--- 0000000000023000 0fd:00000 libk5crypto.so.3.1 95. 00000038bce00000 8 r-x-- 0000000000000000 0fd:00000 libutil-2.5.so 96. 00000038bce02000 2044 ----- 0000000000002000 0fd:00000 libutil-2.5.so 97. 00000038bd001000 4 r---- 0000000000001000 0fd:00000 libutil-2.5.so 98. 00000038bd002000 4 rw--- 0000000000002000 0fd:00000 libutil-2.5.so 99. 00000038be200000 1024 r-x-- 0000000000000000 0fd:00000 libpython2.4.so.1.0 100. 00000038be300000 2048 ----- 0000000000100000 0fd:00000 libpython2.4.so.1.0 101. 00000038be500000 180 rw--- 0000000000100000 0fd:00000 libpython2.4.so.1.0 102. 00000038be52d000 20 rw--- 00000038be52d000 000:00000 [ anon ] 103. 00002aaaaaaab000 21840 rw--- 00002aaaaaaab000 000:00000 [ anon ] 104. 00002aaaac000000 132 rw--- 00002aaaac000000 000:00000 [ anon ] 105. 00002aaaac021000 65404 ----- 00002aaaac021000 000:00000 [ anon ] 106. 00002aaab0000000 2880960 rw--- 00002aaab0000000 000:00000 [ anon ] 107. 00002aab5fd71000 2103728 rw--- 00002aab5fd71000 000:00000 [ anon ] 108. 00002aabe03de000 916436 rw--- 00002aabe03de000 000:00000 [ anon ] 109. 00002aac18311000 3342620 rw--- 00002aac18311000 000:00000 [ anon ] 110. 00002aace4396000 971244 rw--- 00002aace4396000 000:00000 [ anon ] 111. 00002aad1f812000 262856 rw--- 00002aad1f812000 000:00000 [ anon ] 112. 00002aad2f943000 676568 rw--- 00002aad2f943000 000:00000 [ anon ] 113. 00002aad58dfa000 65584 rw--- 00002aad58dfa000 000:00000 [ anon ] 114. 00002af42ab49000 4 rw--- 00002af42ab49000 000:00000 [ anon ] 115. 00002af42ab5d000 276 rw--- 00002af42ab5d000 000:00000 [ anon ] 116. 00002af42aba3000 260 rw--- 00002af42aba3000 000:00000 [ anon ] 117. 00002af42ac16000 55144 r---- 0000000000000000 0fd:00000 locale-archive 118. 00002af42e1f1000 520 rw--- 00002af42e1f1000 000:00000 [ anon ] 119. 00002af42e273000 12 r-x-- 0000000000000000 0fd:00000 timemodule.so 120. 00002af42e276000 2048 ----- 0000000000003000 0fd:00000 timemodule.so 121. 00002af42e476000 8 rw--- 0000000000003000 0fd:00000 timemodule.so 122. 00002af42e478000 20 r-x-- 0000000000000000 0fd:00000 stropmodule.so 123. 00002af42e47d000 2044 ----- 0000000000005000 0fd:00000 stropmodule.so 124. 00002af42e67c000 8 rw--- 0000000000004000 0fd:00000 stropmodule.so 125. 00002af42e67e000 12 r-x-- 0000000000000000 0fd:00000 cStringIO.so 126. 00002af42e681000 2048 ----- 0000000000003000 0fd:00000 cStringIO.so 127. 00002af42e881000 8 rw--- 0000000000003000 0fd:00000 cStringIO.so 128. 00002af42e883000 16 r-x-- 0000000000000000 0fd:00000 collectionsmodule.so 129. 00002af42e887000 2048 ----- 0000000000004000 0fd:00000 collectionsmodule.so 130. 00002af42ea87000 4 rw--- 0000000000004000 0fd:00000 collectionsmodule.so 131. 00002af42ea88000 40 r-x-- 0000000000000000 0fd:00000 _socketmodule.so 132. 00002af42ea92000 2044 ----- 000000000000a000 0fd:00000 _socketmodule.so 133. 00002af42ec91000 16 rw--- 0000000000009000 0fd:00000 _socketmodule.so 134. 00002af42ec95000 16 r-x-- 0000000000000000 0fd:00000 _ssl.so 135. 00002af42ec99000 2044 ----- 0000000000004000 0fd:00000 _ssl.so 136. 00002af42ee98000 4 rw--- 0000000000003000 0fd:00000 _ssl.so 137. 00002af42ee99000 12 r-x-- 0000000000000000 0fd:00000 mathmodule.so 138. 00002af42ee9c000 2048 ----- 0000000000003000 0fd:00000 mathmodule.so 139. 00002af42f09c000 4 rw--- 0000000000003000 0fd:00000 mathmodule.so 140. 00002af42f09d000 20 r-x-- 0000000000000000 0fd:00000 binascii.so 141. 00002af42f0a2000 2044 ----- 0000000000005000 0fd:00000 binascii.so 142. 00002af42f2a1000 4 rw--- 0000000000004000 0fd:00000 binascii.so 143. 00002af42f2a2000 12 r-x-- 0000000000000000 0fd:00000 _randommodule.so 144. 00002af42f2a5000 2044 ----- 0000000000003000 0fd:00000 _randommodule.so 145. 00002af42f4a4000 4 rw--- 0000000000002000 0fd:00000 _randommodule.so 146. 00002af42f4a5000 12 r-x-- 0000000000000000 0fd:00000 fcntlmodule.so 147. 00002af42f4a8000 2044 ----- 0000000000003000 0fd:00000 fcntlmodule.so 148. 00002af42f6a7000 8 rw--- 0000000000002000 0fd:00000 fcntlmodule.so 149. 00002af42f6aa000 260 rw--- 00002af42f6aa000 000:00000 [ anon ] 150. 00002af42f6eb000 20 r-x-- 0000000000000000 0fd:00000 structmodule.so 151. 00002af42f6f0000 2044 ----- 0000000000005000 0fd:00000 structmodule.so 152. 00002af42f8ef000 8 rw--- 0000000000004000 0fd:00000 structmodule.so 153. 00002af42f8f1000 60 r-x-- 0000000000000000 0fd:00000 datetime.so 154. 00002af42f900000 2044 ----- 000000000000f000 0fd:00000 datetime.so 155. 00002af42faff000 16 rw--- 000000000000e000 0fd:00000 datetime.so 156. 00002af42fb03000 260 rw--- 00002af42fb03000 000:00000 [ anon ] 157. 00002af42fb44000 68 r-x-- 0000000000000000 0fd:00000 cPickle.so 158. 00002af42fb55000 2048 ----- 0000000000011000 0fd:00000 cPickle.so 159. 00002af42fd55000 4 rw--- 0000000000011000 0fd:00000 cPickle.so 160. 00002af42fd57000 520 rw--- 00002af42fd57000 000:00000 [ anon ] 161. 00002af42fdda000 1032 rw--- 00002af42fdda000 000:00000 [ anon ] 162. 00002af42fedc000 8 r-x-- 0000000000000000 0fd:00000 grpmodule.so 163. 00002af42fede000 2044 ----- 0000000000002000 0fd:00000 grpmodule.so 164. 00002af4300dd000 4 rw--- 0000000000001000 0fd:00000 grpmodule.so 165. 00002af4300de000 16 r-x-- 0000000000000000 0fd:00000 zlibmodule.so 166. 00002af4300e2000 2044 ----- 0000000000004000 0fd:00000 zlibmodule.so 167. 00002af4302e1000 8 rw--- 0000000000003000 0fd:00000 zlibmodule.so 168. 00002af4302e3000 12 r-x-- 0000000000000000 0fd:00000 selectmodule.so 169. 00002af4302e6000 2048 ----- 0000000000003000 0fd:00000 selectmodule.so 170. 00002af4304e6000 4 rw--- 0000000000003000 0fd:00000 selectmodule.so 171. 00002af4304e8000 264 rw--- 00002af4304e8000 000:00000 [ anon ] 172. 00002af43053d000 40 r-x-- 0000000000000000 0fd:00000 libnss_files-2.5.so 173. 00002af430547000 2044 ----- 000000000000a000 0fd:00000 libnss_files-2.5.so 174. 00002af430746000 4 r---- 0000000000009000 0fd:00000 libnss_files-2.5.so 175. 00002af430747000 4 rw--- 000000000000a000 0fd:00000 libnss_files-2.5.so 176. 00002af430748000 16 r-x-- 0000000000000000 0fd:00000 libnss_dns-2.5.so 177. 00002af43074c000 2044 ----- 0000000000004000 0fd:00000 libnss_dns-2.5.so 178. 00002af43094b000 4 r---- 0000000000003000 0fd:00000 libnss_dns-2.5.so 179. 00002af43094c000 4 rw--- 0000000000004000 0fd:00000 libnss_dns-2.5.so 180. 00007fff2f2a0000 84 rw--- 00007ffffffea000 000:00000 [ stack ] 181. ffffffffff600000 8192 ----- 0000000000000000 000:00000 [ anon ] 182. mapped: 12421472K writeable/private: 12212588K shared: 0K
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6700 root 15 0 11.8g 10g 1940 S 0.7 93.2 1590:43 condor_configd I'd note that the condor_configd typically uses a vanishingly small amount of CPU. Some abnormal event must have occurred for it to accumulate so much TIME.