Bug 1260963
| Summary: | goferd accumulates memory when failing to connect to qdrouterd, segfaulting at the end | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Pavel Moravec <pmoravec> | ||||
| Component: | katello-agent | Assignee: | Katello Bug Bin <katello-bugs> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Tazim Kolhar <tkolhar> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 6.1.1 | CC: | aladke, bbuckingham, bkearney, cdonnell, chenders, chorn, chrobert, cwelton, daniele, daobrien, ddevra, gsim, iboverma, jlyle, jsherril, kgiusti, khuh, mmccune, mwade, pmoravec, sauchter, sthirugn, tkolhar, tross | ||||
| Target Milestone: | Unspecified | Keywords: | Triaged | ||||
| Target Release: | Unused | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | qpid-proton-0.9-11 | Doc Type: | Bug Fix | ||||
| Doc Text: |
Client systems would have issues connecting to Satellite due to out of memory errors from the qpid subsystem.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-01-21 07:51:25 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Valgrind details about the memory accumulation (reordered from the biggest "memory consumers"): ==25650== 7,406,472 bytes in 339 blocks are possibly lost in loss record 9,376 of 9,376 ==25650== at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25650== by 0xFF6C2D2: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0xFFDB8A3: BUF_MEM_grow (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0xFCC675D: ssl23_connect (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFCC7181: ssl23_write (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFCE5F67: ??? (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFFDC02A: BIO_write (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0x229A13B3: ??? (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x229953BB: pn_transport_pending (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x22998AE6: ??? (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x22998B36: ??? (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x22999112: pn_reactor_selectable_transport (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== ==25650== 5,901,312 bytes in 339 blocks are possibly lost in loss record 9,375 of 9,376 ==25650== at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25650== by 0xFF6C2D2: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0xFFE3B24: ??? (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0xFFE3C61: BIO_new_bio_pair (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0x229A10BD: ??? (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x227297A2: ??? (in /usr/lib64/python2.7/site-packages/_cproton.so) ==25650== by 0x4F15B93: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4F171AC: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4EA4097: ??? (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4E7F072: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4ED19EC: ??? (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4ED4E32: ??? (in /usr/lib64/libpython2.7.so.1.0) ==25650== ==25650== 5,901,312 bytes in 339 blocks are possibly lost in loss record 9,374 of 9,376 ==25650== at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25650== by 0xFF6C2D2: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0xFFE3B67: ??? (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0xFFE3C61: BIO_new_bio_pair (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0x229A10BD: ??? (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x227297A2: ??? (in /usr/lib64/python2.7/site-packages/_cproton.so) ==25650== by 0x4F15B93: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4F171AC: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4EA4097: ??? (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4E7F072: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4ED19EC: ??? (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4ED4E32: ??? (in /usr/lib64/libpython2.7.so.1.0) ==25650== ==25650== 5,665,368 bytes in 339 blocks are possibly lost in loss record 9,373 of 9,376 ==25650== at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25650== by 0xFF6C2D2: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0xFCC4112: ssl3_setup_read_buffer (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFCC4278: ssl3_setup_buffers (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFCC60D0: ssl23_connect (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFCC7181: ssl23_write (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFCE5F67: ??? (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFFDC02A: BIO_write (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0x229A13B3: ??? (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x229953BB: pn_transport_pending (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x22998AE6: ??? (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x22998B36: ??? (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== ==25650== 5,613,840 bytes in 339 blocks are possibly lost in loss record 9,372 of 9,376 ==25650== at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25650== by 0xFF6C2D2: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0xFCC421E: ssl3_setup_write_buffer (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFCC4287: ssl3_setup_buffers (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFCC60D0: ssl23_connect (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFCC7181: ssl23_write (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFCE5F67: ??? (in /usr/lib64/libssl.so.1.0.1e) ==25650== by 0xFFDC02A: BIO_write (in /usr/lib64/libcrypto.so.1.0.1e) ==25650== by 0x229A13B3: ??? (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x229953BB: pn_transport_pending (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x22998AE6: ??? (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x22998B36: ??? (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== ==25650== 5,554,176 bytes in 339 blocks are possibly lost in loss record 9,371 of 9,376 ==25650== at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==25650== by 0x229905AD: pn_transport (in /usr/lib64/libqpid-proton.so.2.0.0) ==25650== by 0x2270C157: ??? (in /usr/lib64/python2.7/site-packages/_cproton.so) ==25650== by 0x4F15B93: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4F171AC: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4EA4097: ??? (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4E7F072: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4E8E084: ??? (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4E7F072: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4F1334B: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4F171AC: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==25650== by 0x4EA4097: ??? (in /usr/lib64/libpython2.7.so.1.0) ==25650== .. Created attachment 1071299 [details]
coredump
Coredump. (need to have debuginfo for python-2.7.5-16.el7.x86_64 and qpid-proton-0.9-4.el7.x86_64)
bt shows:
(gdb) frame 0
#0 pn_reactor_attachments (reactor=reactor@entry=0x0) at /usr/src/debug/qpid-proton-0.9/proton-c/src/reactor/reactor.c:119
119 return reactor->attachments;
(gdb) list
114 }
115 }
116
117 pn_record_t *pn_reactor_attachments(pn_reactor_t *reactor) {
118 assert(reactor);
119 return reactor->attachments;
120 }
121
122 pn_millis_t pn_reactor_get_timeout(pn_reactor_t *reactor) {
123 assert(reactor);
(gdb) p reactor
$6 = (pn_reactor_t *) 0x0
(gdb)
Why the assert on previous line did not work? It cant be due to some concurrent change / memory access rewriting the pointer to null, since:
(gdb) frame 1
#1 0x00007f3a0014b2e3 in _wrap_pn_reactor_attachments (self=<optimized out>, args=<optimized out>)
at /usr/src/debug/qpid-proton-0.9/proton-c/bindings/python/cprotonPYTHON_wrap.c:23077
23077 result = (pn_record_t *)pn_reactor_attachments(arg1);
(gdb) p arg1
$7 = (pn_reactor_t *) 0x0
(gdb)
Can't be the cause some limit was reached (i.e. file descriptors, some AMQP connection/session/whatever pool exhausted)?
Btw. when changing ulimit -n (both to lower values or to higher), goferd does _not_ segfault. It just consumes more and more memory. Segfault is present only for ulimit -n 1024 (In reply to Pavel Moravec from comment #4) > Btw. when changing ulimit -n (both to lower values or to higher), goferd > does _not_ segfault. It just consumes more and more memory. Segfault is > present only for ulimit -n 1024 I was wrong / too eager. For ulimit -n approx. 900 or lower, memory growth stops after a while and no segfault For ulimit -n 900 - 1024, memory growth until 86MB and segfault For ulimit -n 1100 or more, memory growth until some higher limit (i.e. 90MB for 1100) or OOM killer. The memory growth and segfault seems to be related to reconnect loop. Since when qdrouterd is started for a while and let goferd to connect to itself (until it is killed again), this resets the above scenario - finally stable memory usage starts to grow again. (In reply to Pavel Moravec from comment #5) > For ulimit -n approx. 900 or lower, memory growth stops after a while and no > segfault > For ulimit -n 900 - 1024, memory growth until 86MB and segfault > For ulimit -n 1100 or more, memory growth until some higher limit (i.e. 90MB > for 1100) or OOM killer. Might be unrelated, but this reminds me of bz1242406 . Hi Pavel, couple of quick points: re: assert - that assert statement is removed by the compiler for release builds (NDEBUG is defined in that case) re: leaks - these leaks look as if they are related to initializing an SSL connection, which is held by the pn_transport_t object. Likely this indicates the leak involves the connection teardown process. Standalone reproducer using proton reactor & proton only:
(assuming nothing listens on port 12345 on localhost):
(assuming some valid SSL certs/CA are provided to ca_certificate / client_certificate)
#!/usr/bin/python
from time import sleep
from proton import ConnectionException
from proton import SSLDomain, SSLException
from proton.utils import BlockingConnection
import traceback
ca_certificate='/etc/rhsm/ca/katello-default-ca.pem'
client_certificate='/etc/pki/consumer/bundle.pem'
client_key=None
domain = SSLDomain(SSLDomain.MODE_CLIENT)
domain.set_trusted_ca_db(ca_certificate)
domain.set_credentials(
client_certificate,
client_key or client_certificate, None)
domain.set_peer_authentication(SSLDomain.VERIFY_PEER)
while True:
sleep(0.1)
try:
conn = BlockingConnection("amqps://localhost:12345", ssl_domain=domain, heartbeat=2)
except ConnectionException, e:
print e
print(traceback.format_exc())
I've committed two fixes upstream that I believe address the leaks: https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;h=c799a29 https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;h=bbba61a A fix is available upstream: https://issues.apache.org/jira/browse/PROTON-1028 (In reply to Gordon Sim from comment #11) > I've committed two fixes upstream that I believe address the leaks: > > https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;h=c799a29 > https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;h=bbba61a I can confirm backporting the them fixes both the standalone reproducer and also goferd reproducer. Thanks! For backporting attempt: please apply this patch _after_ applying patch for bz1272596 that makes changes in very adjacent areas. Moving this to 6.1.z I don't see any reason here why it shouldn't be. If for some reason the patches cannot be backported we can punt. VERIFIED: # rpm -qa | grep foreman ruby193-rubygem-foreman_discovery-2.0.0.23-1.el7sat.noarch cloud-qe-6.idmqe.lab.eng.bos.redhat.com-foreman-client-1.0-1.noarch cloud-qe-6.idmqe.lab.eng.bos.redhat.com-foreman-proxy-client-1.0-1.noarch foreman-gce-1.7.2.50-1.el7sat.noarch rubygem-hammer_cli_foreman-0.1.4.15-1.el7sat.noarch foreman-1.7.2.50-1.el7sat.noarch ruby193-rubygem-foreman_docker-1.2.0.24-1.el7sat.noarch ruby193-rubygem-foreman-tasks-0.6.15.7-1.el7sat.noarch rubygem-hammer_cli_foreman_bootdisk-0.1.2.7-1.el7sat.noarch rubygem-hammer_cli_foreman_docker-0.0.3.10-1.el7sat.noarch foreman-selinux-1.7.2.17-1.el7sat.noarch foreman-libvirt-1.7.2.50-1.el7sat.noarch foreman-compute-1.7.2.50-1.el7sat.noarch foreman-ovirt-1.7.2.50-1.el7sat.noarch ruby193-rubygem-foreman-redhat_access-0.2.4-1.el7sat.noarch rubygem-hammer_cli_foreman_discovery-0.0.1.10-1.el7sat.noarch ruby193-rubygem-foreman_bootdisk-4.0.2.14-1.el7sat.noarch foreman-vmware-1.7.2.50-1.el7sat.noarch rubygem-hammer_cli_foreman_tasks-0.0.3.5-1.el7sat.noarch foreman-debug-1.7.2.50-1.el7sat.noarch foreman-proxy-1.7.2.8-1.el7sat.noarch ruby193-rubygem-foreman_hooks-0.3.7-2.el7sat.noarch ruby193-rubygem-foreman_gutterball-0.0.1.9-1.el7sat.noarch foreman-postgresql-1.7.2.50-1.el7sat.noarch cloud-qe-6.idmqe.lab.eng.bos.redhat.com-foreman-proxy-1.0-1.noarch steps: 1. setting DELAY = 1 and MAX_DELAY = 1 in /usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/connection.py 2. (on Satellite) service qdrouterd stop 3. (on Capsule) while true; do ps aux | grep python | grep gofer; sleep 5; done ]# while true; do ps aux | grep python | grep gofer; sleep 5; done root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:04 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.4 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:05 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground root 8189 39.3 0.2 1939920 44804 ? Ssl Jan08 5604:06 python /usr/bin/goferd --foreground As, per my understanding, there is no memory accumulation and goferd process seems to be constant Also, as I was monitoring /var/log/messages on both sat6 and capsule did not face any seg fault issue moving it to verified ... Thanks !! Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:0051 |
Description of problem: When goferd cant connect to qdrouterd due to whatever reason, it's memory consumption grows over the time. Once it reaches approx. 86MB (per ps output, RSS column), goferd segfaults (abrt does not generate a report, though). Version-Release number of selected component (if applicable): python-gofer-2.6.2-2.el7sat.noarch gofer-2.6.2-2.el7sat.noarch python-gofer-qpid-2.6.2-2.el7sat.noarch python-gofer-proton-2.6.2-2.el7sat.noarch How reproducible: 100% Steps to Reproduce: 1. (just for faster reproducer, lower delays between connection probes by setting DELAY = 1 and MAX_DELAY = 1 in /usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/connection.py) 2. (on Satellite) service qdrouterd stop 3. (on Capsule) while true; do ps aux | grep python | grep gofer; sleep 5; done Actual results: - RSS of goferd process growing over time - once it reaches approx 86MB, PID of the process changes in the ps outputs and /var/log/messages has: Sep 8 11:05:05 my-capsule kernel: python[24968]: segfault at 0 ip 00007f2e400f5e70 sp 00007f2e4fffab48 error 4 in libqpid-proton.so.2.0.0[7f2e400c6000+4c000] Expected results: - no memory accumulation - no segfault Additional info: running goferd under valgrind, it shows some "definitely lost" bytes. But that's some constant occurring regardless of number of failing connection attempts. segfault sounds to be caused by proton bug, as coredump backtrace starts with: #0 pn_reactor_attachments (reactor=reactor@entry=0x0) at /usr/src/debug/qpid-proton-0.9/proton-c/src/reactor/reactor.c:119 #1 0x00007f3a0014b2e3 in _wrap_pn_reactor_attachments (self=<optimized out>, args=<optimized out>) at /usr/src/debug/qpid-proton-0.9/proton-c/bindings/python/cprotonPYTHON_wrap.c:23077 .. (will check this later on)