Bug 713738
Summary: | WSGI crash with "double free or corruption" message | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Update Infrastructure for Cloud Providers | Reporter: | Sachin Ghai <sghai> | ||||
Component: | Upstream | Assignee: | Jay Dobies <jason.dobies> | ||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | wes hayutin <whayutin> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 2.0 | CC: | kbidarka, sghai, tsanders | ||||
Target Milestone: | --- | ||||||
Target Release: | --- | ||||||
Hardware: | x86_64 | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2012-05-31 12:51:55 UTC | Type: | --- | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | 696669 | ||||||
Bug Blocks: | |||||||
Attachments: |
|
Added this to the tracking bug we have for WSGI crashes: 677735 Facing this issue in new builds (pulp 0.194 and rhui-tools 2.0.32) <snippet> *** glibc detected *** (wsgi:pulp) : corrupted double-linked list: 0x00007fa82c1f2ff0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x75676)[0x7fa8637e3676] /lib64/libc.so.6(+0x7815a)[0x7fa8637e615a] /usr/lib64/libnsspem.so(+0x18e72)[0x7fa83c47fe72] /usr/lib64/libnsspem.so(+0xa4f4)[0x7fa83c4714f4] /usr/lib64/libnsspem.so(+0xa64d)[0x7fa83c47164d] /usr/lib64/libnsspem.so(+0x17e0f)[0x7fa83c47ee0f] /usr/lib64/libnsspem.so(+0x13df1)[0x7fa83c47adf1] /usr/lib64/libnss3.so(PK11_Sign+0xb1)[0x7fa855518b71] /usr/lib64/libssl3.so(+0xe620)[0x7fa8523eb620] /usr/lib64/libssl3.so(+0xf159)[0x7fa8523ec159] /usr/lib64/libssl3.so(+0x12860)[0x7fa8523ef860] /usr/lib64/libssl3.so(+0x13e30)[0x7fa8523f0e30] /usr/lib64/libssl3.so(+0x148cc)[0x7fa8523f18cc] /usr/lib64/libssl3.so(+0x17169)[0x7fa8523f4169] /usr/lib64/libssl3.so(+0x1cf15)[0x7fa8523f9f15] /usr/lib64/libssl3.so(SSL_ForceHandshake+0x9f)[0x7fa8523fb67f] /usr/lib64/libcurl.so.4(Curl_nss_connect+0xb70)[0x7fa852883e10] /usr/lib64/libcurl.so.4(Curl_ssl_connect+0x2b)[0x7fa85287b34b] /usr/lib64/libcurl.so.4(Curl_http_connect+0x5b)[0x7fa852859e5b] /usr/lib64/libcurl.so.4(Curl_protocol_connect+0xd2)[0x7fa852860612] /usr/lib64/libcurl.so.4(Curl_connect+0x2fc)[0x7fa852866abc] /usr/lib64/libcurl.so.4(Curl_perform+0x80)[0x7fa85286ea50] /usr/lib64/python2.6/site-packages/pycurl.so(+0x873b)[0x7fa852a9e73b] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x521e)[0x7fa85a4f081e] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x904)[0x7fa85a4f25a4] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x501e)[0x7fa85a4f061e] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5f2d)[0x7fa85a4f152d] ======= Memory map: ======== 7fa8105d4000-7fa8105d5000 ---p 00000000 00:00 0 7fa8105d5000-7fa810fd5000 rw-p 00000000 00:00 0 7fa810fe8000-7fa810fe9000 ---p 00000000 00:00 0 7fa810fe9000-7fa8119e9000 rw-p 00000000 00:00 0 7fa8119e9000-7fa8119ea000 ---p 00000000 00:00 0 7fa8119ea000-7fa8123ea000 rw-p 00000000 00:00 0 7fa8123ea000-7fa8123eb000 ---p 00000000 00:00 0 7fa8123eb000-7fa812deb000 rw-p 00000000 00:00 0 <snippet> I got the same issue with new build.. pulp 0.208 rh-rhui-tools 2.0.40. /var/log/httpd/error_log ============================ [Thu Jul 21 18:50:43 2011] [error] 2011-07-21 18:50:43,795 pulp.server.async:ERROR: Task [3cd48ecf-b39c-11e0-b712-5254006c2658] Task 3cd48ecf-b39c-11e0-b712-5254006c2658: CdsApi.redistribute(rhel-6-server-rhui-rpms-6.0-i386, ) scheduled to run immediately conflicts with [3cbb4963-b39c-11e0-b66d-5254006c2658] Task 3cbb4963-b39c-11e0-b66d-5254006c2658: CdsApi.redistribute(rhel-6-server-rhui-rpms-6.0-i386, ) scheduled to run immediately and cannot be enqueued *** glibc detected *** (wsgi:pulp) : double free or corruption (!prev): 0x00007f92289750a0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x3baf875716)[0x7f926a537716] /usr/lib64/libnsspem.so(+0x18e72)[0x7f9244689e72] /usr/lib64/libnsspem.so(+0xa4f4)[0x7f924467b4f4] /usr/lib64/libnsspem.so(+0xa64d)[0x7f924467b64d] /usr/lib64/libnsspem.so(+0x17e0f)[0x7f9244688e0f] /usr/lib64/libnsspem.so(+0x13df1)[0x7f9244684df1] /usr/lib64/libnss3.so(PK11_Sign+0xb1)[0x7f926745ab71] /usr/lib64/libssl3.so(+0x3bb4c0e620)[0x7f9267989620] /usr/lib64/libssl3.so(+0x3bb4c0f159)[0x7f926798a159] /usr/lib64/libssl3.so(+0x3bb4c12860)[0x7f926798d860] /usr/lib64/libssl3.so(+0x3bb4c13e30)[0x7f926798ee30] /usr/lib64/libssl3.so(+0x3bb4c148cc)[0x7f926798f8cc] /usr/lib64/libssl3.so(+0x3bb4c17169)[0x7f9267992169] /usr/lib64/libssl3.so(+0x3bb4c1cf15)[0x7f9267997f15] /usr/lib64/libssl3.so(SSL_ForceHandshake+0x9f)[0x7f926799967f] /usr/lib64/libcurl.so.4(Curl_nss_connect+0xb70)[0x7f925907be10] /usr/lib64/libcurl.so.4(Curl_ssl_connect+0x2b)[0x7f925907334b] /usr/lib64/libcurl.so.4(Curl_http_connect+0x5b)[0x7f9259051e5b] /usr/lib64/libcurl.so.4(Curl_protocol_connect+0xd2)[0x7f9259058612] /usr/lib64/libcurl.so.4(Curl_connect+0x2fc)[0x7f925905eabc] /usr/lib64/libcurl.so.4(Curl_perform+0x80)[0x7f9259066a50] /usr/lib64/python2.6/site-packages/pycurl.so(+0x864b)[0x7f925929664b] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5176)[0x7f925ffffad6] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x92c)[0x7f926000181c] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5304)[0x7f925ffffc64] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x63ef)[0x7f9260000d4f] ======= Memory map: ======== 7f9204000000-7f920438f000 rw-p 00000000 00:00 0 7f920438f000-7f9208000000 ---p 00000000 00:00 0 7f920c000000-7f9210000000 rw-p 00000000 00:00 0 7f9216deb000-7f9216dec000 ---p 00000000 00:00 0 7f9216dec000-7f92177ec000 rw-p 00000000 00:00 0 7f92181ed000-7f92181ee000 ---p 00000000 00:00 0 7f92181ee000-7f9218bee000 rw-p 00000000 00:00 0 7f921c7f4000-7f921c7f5000 ---p 00000000 00:00 0 7f921c7f5000-7f921d1f5000 rw-p 00000000 00:00 0 7f921d8f5000-7f921dbf6000 rw-p 00000000 00:00 0 7f921dbf6000-7f921dbf7000 ---p 00000000 00:00 0 7f921dbf7000-7f921e5f7000 rw-p 00000000 00:00 0 7f921e5f7000-7f921e5f8000 ---p 00000000 00:00 0 7f921e5f8000-7f921eff8000 rw-p 00000000 00:00 0 7f921eff8000-7f921eff9000 ---p 00000000 00:00 0 8000,1 71% I was checking the apache's error_log to investigate the defect 723534. then I found these logs. Please check this on the latest ISO (Pulp .211, Grinder 0.108). We need reproduction steps if they are possible, or at least to understand how frequently this occurs. I meant to move to ON_QA, not NEEDINFO. Yesterday, I installed a fresh setup with latest ISO having pulp build 0.211 and Grinder 0.108. I'm not sure about the exact steps to reproduce this issue.But I started with repo sync from CDN to RHUA. I started the sync for multiple repos. Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server (RPMs) (5Server-x86_64) 07-27-2011 12:35 07-27-2011 09:25 Success Red Hat Enterprise Linux Server 6 Optional Updates (RPMs) (6Server-x86_64) 07-27-2011 12:12 07-27-2011 06:43 Success Red Hat Enterprise Linux Server 6 Releases (RPMs) (6Server-x86_64) 07-27-2011 12:12 07-27-2011 09:19 Success Red Hat Enterprise Linux Server 6 Updates (RPMs) (6Server-x86_64) 07-27-2011 12:13 07-27-2011 06:44 Success Red Hat Update Infrastructure 1.2 (RPMs) (5Server-i386) 07-27-2011 16:48 07-27-2011 10:49 Success Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64) 07-27-2011 16:48 07-27-2011 10:49 Success Connected: dhcp201-127.englab.pnq.redhat.com This has downloaded approximate 22,000 packages on my local setup without any error in apache error_log. I tested this issue on Amazon portal with latest build 0.214 This is not reproducible with new builds. Added following repos, and sync ran successfully (from CDN to RHUA). Also one of the CDS node is also successfully synchronized. -= Repository Synchronization Status =- Last Refreshed: 05:01:55 (updated every 5 seconds, ctrl+c to exit) Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server (RPMs) (5Server-x86_64) 07-27-2011 10:09 07-27-2011 04:56 Success Red Hat Enterprise Linux Server 6 Optional Releases (RPMs) (6Server-x86_64) 07-27-2011 10:07 07-27-2011 04:22 Success Red Hat Enterprise Linux Server 6 Optional Updates (RPMs) (6Server-x86_64) 07-27-2011 10:07 07-27-2011 04:25 Success Red Hat Enterprise Linux Server 6 Updates (RPMs) (6Server-x86_64) 07-27-2011 10:07 07-27-2011 04:32 Success Connected: ip-10-86-222-208.ec2.internal -= CDS Synchronization Status =- Last Refreshed: 06:08:28 (updated every 5 seconds, ctrl+c to exit) cds109 ...................................................... [ UP ] cds52 ....................................................... [ UP ] Next Sync Last Sync Last Result ------------------------------------------------------------------------------ cds109 07-27-2011 10:20 07-27-2011 05:41 Success cds52 In Progress Never Never Connected: ip-10-86-222-208.ec2.internal I verified this issue locally as well as on amazon and it is not reproducible with new builds ( pulp 0.211/0.214 and rh-rhui-tools 2.0.41). moving to release pending closing out, product released |
Created attachment 505031 [details] apache's error_log file. Description of problem: I scheduled repo sync for multiple repos. Sync started and was runing fine for few minutes and after few minutes sync status changed from running to scheduled. ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= Repository Synchronization Status =- Last Refreshed: 17:18:42 (updated every 5 seconds, ctrl+c to exit) Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server 6 Optional Releases (RPMs) (6Server-x86_64) 06-16-2011 17:18 Never scheduled Red Hat Enterprise Linux Server 6 Optional Updates (RPMs) (6Server-x86_64) 06-16-2011 18:18 Never running Red Hat Update Infrastructure 1.2 (RPMs) (5Server-i386) 06-16-2011 17:18 06-16-2011 16:37 running Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64) 06-16-2011 17:18 06-16-2011 16:37 running Connected: dhcp193-163.pnq.redhat.com ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= Repository Synchronization Status =- Last Refreshed: 17:19:14 (updated every 5 seconds, ctrl+c to exit) Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server 6 Optional Releases (RPMs) (6Server-x86_64) 06-16-2011 18:18 Never running Red Hat Enterprise Linux Server 6 Optional Updates (RPMs) (6Server-x86_64) 06-16-2011 18:18 Never running Red Hat Update Infrastructure 1.2 (RPMs) (5Server-i386) 06-16-2011 18:18 06-16-2011 17:19 finished Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64) 06-16-2011 18:18 06-16-2011 17:19 finished ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= Repository Synchronization Status =- Last Refreshed: 18:05:16 (updated every 5 seconds, ctrl+c to exit) Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server 6 Optional Releases (RPMs) (6Server-x86_64) 06-16-2011 18:18 Never scheduled Red Hat Enterprise Linux Server 6 Optional Updates (RPMs) (6Server-x86_64) 06-16-2011 18:18 Never scheduled Red Hat Update Infrastructure 1.2 (RPMs) (5Server-i386) 06-16-2011 18:18 06-16-2011 17:19 scheduled Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64) 06-16-2011 18:18 06-16-2011 17:19 scheduled Connected: dhcp193-163.pnq.redhat.com ------------------------------------------------------------------------------ And I got a long backtrace in /var/log/httpd/error_log. [Thu Jun 16 17:50:55 2011] [error] import md5 *** glibc detected *** (wsgi:pulp) : double free or corruption (!prev): 0x00007fa1e17a8d10 *** ======= Backtrace: ========= The complete back trace in attachment tail -f /var/log/pulp/pulp.log ================================ 2011-06-16 18:02:43,252 [INFO][MainThread] start() @ async.py:331 - Task reply handler, started. 2011-06-16 18:02:43,253 [INFO][asynctaskreplyqueue] connect() @ broker.py:99 - connecting: {dhcp193-163.pnq.redhat.com:5674}: transport=SSL host=dhcp193-163.pnq.redhat.com port=5674 cacert=/etc/pki/rhua/qpid-ca.crt clientcert=/etc/pki/rhua/qpid-client.crt Version-Release number of selected component (if applicable): pulp 0.190 rhui-tools 2.0.30 How reproducible: always Steps to Reproduce: 1. Scheduled repo sync for multiple repos 2. when sync starts, tail the apache's error_log file Actual results: Long backtrace as in attachment. Expected results: Shouldn't be any traceback and backtrace in log file Additional info: