713738 – WSGI crash with "double free or corruption" message

Bug 713738 - WSGI crash with "double free or corruption" message

Summary: WSGI crash with "double free or corruption" message

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Update Infrastructure for Cloud Providers
Classification:	Red Hat
Component:	Upstream
Sub Component:
Version:	2.0
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Jay Dobies
QA Contact:	wes hayutin
Docs Contact:
URL:
Whiteboard:
Depends On:	696669
Blocks:
TreeView+	depends on / blocked

Reported:	2011-06-16 12:48 UTC by Sachin Ghai
Modified:	2012-05-31 12:51 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2012-05-31 12:51:55 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
apache's error_log file. (68.46 KB, application/octet-stream) 2011-06-16 12:48 UTC, Sachin Ghai	no flags	Details
View All

Description Sachin Ghai 2011-06-16 12:48:01 UTC

Created attachment 505031 [details]
apache's error_log file.

Description of problem:
I scheduled repo sync for multiple repos. Sync started and was runing fine for few minutes and after few minutes sync status changed from running to scheduled.

------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= Repository Synchronization Status =-

Last Refreshed: 17:18:42
(updated every 5 seconds, ctrl+c to exit)

Next Sync                    Last Sync                    Last Result         
------------------------------------------------------------------------------
Red Hat Enterprise Linux Server 6 Optional Releases (RPMs) (6Server-x86_64)
06-16-2011 17:18             Never                        scheduled  

Red Hat Enterprise Linux Server 6 Optional Updates (RPMs) (6Server-x86_64)
06-16-2011 18:18             Never                        running    

Red Hat Update Infrastructure 1.2 (RPMs) (5Server-i386)
06-16-2011 17:18             06-16-2011 16:37             running    

Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64)
06-16-2011 17:18             06-16-2011 16:37             running    


                                         Connected: dhcp193-163.pnq.redhat.com
------------------------------------------------------------------------------




------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= Repository Synchronization Status =-

Last Refreshed: 17:19:14
(updated every 5 seconds, ctrl+c to exit)

Next Sync                    Last Sync                    Last Result         
------------------------------------------------------------------------------
Red Hat Enterprise Linux Server 6 Optional Releases (RPMs) (6Server-x86_64)
06-16-2011 18:18             Never                        running    

Red Hat Enterprise Linux Server 6 Optional Updates (RPMs) (6Server-x86_64)
06-16-2011 18:18             Never                        running    

Red Hat Update Infrastructure 1.2 (RPMs) (5Server-i386)
06-16-2011 18:18             06-16-2011 17:19             finished   

Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64)
06-16-2011 18:18             06-16-2011 17:19             finished   


------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= Repository Synchronization Status =-

Last Refreshed: 18:05:16
(updated every 5 seconds, ctrl+c to exit)

Next Sync                    Last Sync                    Last Result         
------------------------------------------------------------------------------
Red Hat Enterprise Linux Server 6 Optional Releases (RPMs) (6Server-x86_64)
06-16-2011 18:18             Never                        scheduled  

Red Hat Enterprise Linux Server 6 Optional Updates (RPMs) (6Server-x86_64)
06-16-2011 18:18             Never                        scheduled  

Red Hat Update Infrastructure 1.2 (RPMs) (5Server-i386)
06-16-2011 18:18             06-16-2011 17:19             scheduled  

Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64)
06-16-2011 18:18             06-16-2011 17:19             scheduled  


                                         Connected: dhcp193-163.pnq.redhat.com
------------------------------------------------------------------------------




And I got a long backtrace in /var/log/httpd/error_log.

[Thu Jun 16 17:50:55 2011] [error]   import md5
*** glibc detected *** (wsgi:pulp)    : double free or corruption (!prev): 0x00007fa1e17a8d10 ***
======= Backtrace: =========

The complete back trace in attachment


tail -f /var/log/pulp/pulp.log
================================

2011-06-16 18:02:43,252 [INFO][MainThread] start() @ async.py:331 - Task reply handler, started.
2011-06-16 18:02:43,253 [INFO][asynctaskreplyqueue] connect() @ broker.py:99 - connecting:
{dhcp193-163.pnq.redhat.com:5674}:
transport=SSL
host=dhcp193-163.pnq.redhat.com
port=5674
cacert=/etc/pki/rhua/qpid-ca.crt
clientcert=/etc/pki/rhua/qpid-client.crt



Version-Release number of selected component (if applicable):
pulp 0.190
rhui-tools 2.0.30

How reproducible:
always

Steps to Reproduce:
1. Scheduled repo sync for multiple repos
2. when sync starts, tail the apache's error_log file

  
Actual results:
Long backtrace as in attachment.

Expected results:
Shouldn't be any traceback and backtrace in log file

Additional info:

Comment 1 Jay Dobies 2011-06-16 13:02:43 UTC

Added this to the tracking bug we have for WSGI crashes: 677735

Comment 2 Sachin Ghai 2011-06-23 06:04:44 UTC

Facing this issue in new builds (pulp 0.194 and rhui-tools 2.0.32)


<snippet>
 *** glibc detected *** (wsgi:pulp)    : corrupted double-linked list: 0x00007fa82c1f2ff0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75676)[0x7fa8637e3676]
/lib64/libc.so.6(+0x7815a)[0x7fa8637e615a]
/usr/lib64/libnsspem.so(+0x18e72)[0x7fa83c47fe72]
/usr/lib64/libnsspem.so(+0xa4f4)[0x7fa83c4714f4]
/usr/lib64/libnsspem.so(+0xa64d)[0x7fa83c47164d]
/usr/lib64/libnsspem.so(+0x17e0f)[0x7fa83c47ee0f]
/usr/lib64/libnsspem.so(+0x13df1)[0x7fa83c47adf1]
/usr/lib64/libnss3.so(PK11_Sign+0xb1)[0x7fa855518b71]
/usr/lib64/libssl3.so(+0xe620)[0x7fa8523eb620]
/usr/lib64/libssl3.so(+0xf159)[0x7fa8523ec159]
/usr/lib64/libssl3.so(+0x12860)[0x7fa8523ef860]
/usr/lib64/libssl3.so(+0x13e30)[0x7fa8523f0e30]
/usr/lib64/libssl3.so(+0x148cc)[0x7fa8523f18cc]
/usr/lib64/libssl3.so(+0x17169)[0x7fa8523f4169]
/usr/lib64/libssl3.so(+0x1cf15)[0x7fa8523f9f15]
/usr/lib64/libssl3.so(SSL_ForceHandshake+0x9f)[0x7fa8523fb67f]
/usr/lib64/libcurl.so.4(Curl_nss_connect+0xb70)[0x7fa852883e10]
/usr/lib64/libcurl.so.4(Curl_ssl_connect+0x2b)[0x7fa85287b34b]
/usr/lib64/libcurl.so.4(Curl_http_connect+0x5b)[0x7fa852859e5b]
/usr/lib64/libcurl.so.4(Curl_protocol_connect+0xd2)[0x7fa852860612]
/usr/lib64/libcurl.so.4(Curl_connect+0x2fc)[0x7fa852866abc]
/usr/lib64/libcurl.so.4(Curl_perform+0x80)[0x7fa85286ea50]
/usr/lib64/python2.6/site-packages/pycurl.so(+0x873b)[0x7fa852a9e73b]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x521e)[0x7fa85a4f081e]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x904)[0x7fa85a4f25a4]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x501e)[0x7fa85a4f061e]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5f2d)[0x7fa85a4f152d]
======= Memory map: ========
7fa8105d4000-7fa8105d5000 ---p 00000000 00:00 0
7fa8105d5000-7fa810fd5000 rw-p 00000000 00:00 0
7fa810fe8000-7fa810fe9000 ---p 00000000 00:00 0
7fa810fe9000-7fa8119e9000 rw-p 00000000 00:00 0
7fa8119e9000-7fa8119ea000 ---p 00000000 00:00 0
7fa8119ea000-7fa8123ea000 rw-p 00000000 00:00 0
7fa8123ea000-7fa8123eb000 ---p 00000000 00:00 0
7fa8123eb000-7fa812deb000 rw-p 00000000 00:00 0
<snippet>

Comment 3 Sachin Ghai 2011-07-22 13:03:05 UTC

I got the same issue with new build..

pulp 0.208
rh-rhui-tools 2.0.40.

/var/log/httpd/error_log
============================
[Thu Jul 21 18:50:43 2011] [error] 2011-07-21 18:50:43,795 pulp.server.async:ERROR: Task [3cd48ecf-b39c-11e0-b712-5254006c2658] Task 3cd48ecf-b39c-11e0-b712-5254006c2658: CdsApi.redistribute(rhel-6-server-rhui-rpms-6.0-i386, ) scheduled to run immediately conflicts with [3cbb4963-b39c-11e0-b66d-5254006c2658] Task 3cbb4963-b39c-11e0-b66d-5254006c2658: CdsApi.redistribute(rhel-6-server-rhui-rpms-6.0-i386, ) scheduled to run immediately and cannot be enqueued
*** glibc detected *** (wsgi:pulp)    : double free or corruption (!prev): 0x00007f92289750a0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x3baf875716)[0x7f926a537716]
/usr/lib64/libnsspem.so(+0x18e72)[0x7f9244689e72]
/usr/lib64/libnsspem.so(+0xa4f4)[0x7f924467b4f4]
/usr/lib64/libnsspem.so(+0xa64d)[0x7f924467b64d]
/usr/lib64/libnsspem.so(+0x17e0f)[0x7f9244688e0f]
/usr/lib64/libnsspem.so(+0x13df1)[0x7f9244684df1]
/usr/lib64/libnss3.so(PK11_Sign+0xb1)[0x7f926745ab71]
/usr/lib64/libssl3.so(+0x3bb4c0e620)[0x7f9267989620]
/usr/lib64/libssl3.so(+0x3bb4c0f159)[0x7f926798a159]
/usr/lib64/libssl3.so(+0x3bb4c12860)[0x7f926798d860]
/usr/lib64/libssl3.so(+0x3bb4c13e30)[0x7f926798ee30]
/usr/lib64/libssl3.so(+0x3bb4c148cc)[0x7f926798f8cc]
/usr/lib64/libssl3.so(+0x3bb4c17169)[0x7f9267992169]
/usr/lib64/libssl3.so(+0x3bb4c1cf15)[0x7f9267997f15]
/usr/lib64/libssl3.so(SSL_ForceHandshake+0x9f)[0x7f926799967f]
/usr/lib64/libcurl.so.4(Curl_nss_connect+0xb70)[0x7f925907be10]
/usr/lib64/libcurl.so.4(Curl_ssl_connect+0x2b)[0x7f925907334b]
/usr/lib64/libcurl.so.4(Curl_http_connect+0x5b)[0x7f9259051e5b]
/usr/lib64/libcurl.so.4(Curl_protocol_connect+0xd2)[0x7f9259058612]
/usr/lib64/libcurl.so.4(Curl_connect+0x2fc)[0x7f925905eabc]
/usr/lib64/libcurl.so.4(Curl_perform+0x80)[0x7f9259066a50]
/usr/lib64/python2.6/site-packages/pycurl.so(+0x864b)[0x7f925929664b]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5176)[0x7f925ffffad6]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x92c)[0x7f926000181c]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5304)[0x7f925ffffc64]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x63ef)[0x7f9260000d4f]
======= Memory map: ========
7f9204000000-7f920438f000 rw-p 00000000 00:00 0
7f920438f000-7f9208000000 ---p 00000000 00:00 0
7f920c000000-7f9210000000 rw-p 00000000 00:00 0
7f9216deb000-7f9216dec000 ---p 00000000 00:00 0
7f9216dec000-7f92177ec000 rw-p 00000000 00:00 0
7f92181ed000-7f92181ee000 ---p 00000000 00:00 0
7f92181ee000-7f9218bee000 rw-p 00000000 00:00 0
7f921c7f4000-7f921c7f5000 ---p 00000000 00:00 0
7f921c7f5000-7f921d1f5000 rw-p 00000000 00:00 0
7f921d8f5000-7f921dbf6000 rw-p 00000000 00:00 0
7f921dbf6000-7f921dbf7000 ---p 00000000 00:00 0
7f921dbf7000-7f921e5f7000 rw-p 00000000 00:00 0
7f921e5f7000-7f921e5f8000 ---p 00000000 00:00 0
7f921e5f8000-7f921eff8000 rw-p 00000000 00:00 0
7f921eff8000-7f921eff9000 ---p 00000000 00:00 0
                                                                                                         8000,1        71%


I was checking the apache's error_log to investigate the defect 723534. then I found these logs.

Comment 4 Jay Dobies 2011-07-25 19:46:06 UTC

Please check this on the latest ISO (Pulp .211, Grinder 0.108). We need reproduction steps if they are possible, or at least to understand how frequently this occurs.

Comment 5 Jay Dobies 2011-07-25 19:46:28 UTC

I meant to move to ON_QA, not NEEDINFO.

Comment 6 Sachin Ghai 2011-07-27 06:26:02 UTC

Yesterday, I installed a fresh setup with latest ISO having pulp build 0.211 and Grinder 0.108.

I'm not sure about the exact steps to reproduce this issue.But I started with repo sync from CDN to RHUA. I started the sync for multiple repos.



Next Sync                    Last Sync                    Last Result         
------------------------------------------------------------------------------
Red Hat Enterprise Linux Server (RPMs) (5Server-x86_64)
07-27-2011 12:35             07-27-2011 09:25             Success    

Red Hat Enterprise Linux Server 6 Optional Updates (RPMs) (6Server-x86_64)
07-27-2011 12:12             07-27-2011 06:43             Success    

Red Hat Enterprise Linux Server 6 Releases (RPMs) (6Server-x86_64)
07-27-2011 12:12             07-27-2011 09:19             Success    

Red Hat Enterprise Linux Server 6 Updates (RPMs) (6Server-x86_64)
07-27-2011 12:13             07-27-2011 06:44             Success    

Red Hat Update Infrastructure 1.2 (RPMs) (5Server-i386)
07-27-2011 16:48             07-27-2011 10:49             Success    

Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64)
07-27-2011 16:48             07-27-2011 10:49             Success    


                                  Connected: dhcp201-127.englab.pnq.redhat.com


This has downloaded approximate 22,000 packages on my local setup without any error in apache error_log.

Comment 7 Sachin Ghai 2011-07-27 10:14:14 UTC

I tested this issue on Amazon portal with latest build 0.214 This is not reproducible with new builds.

Added following repos, and sync ran successfully (from CDN to RHUA). Also one of the CDS node is also successfully synchronized.


-= Repository Synchronization Status =-

Last Refreshed: 05:01:55
(updated every 5 seconds, ctrl+c to exit)

Next Sync                    Last Sync                    Last Result         
------------------------------------------------------------------------------
Red Hat Enterprise Linux Server (RPMs) (5Server-x86_64)
07-27-2011 10:09             07-27-2011 04:56             Success    

Red Hat Enterprise Linux Server 6 Optional Releases (RPMs) (6Server-x86_64)
07-27-2011 10:07             07-27-2011 04:22             Success    

Red Hat Enterprise Linux Server 6 Optional Updates (RPMs) (6Server-x86_64)
07-27-2011 10:07             07-27-2011 04:25             Success    

Red Hat Enterprise Linux Server 6 Updates (RPMs) (6Server-x86_64)
07-27-2011 10:07             07-27-2011 04:32             Success    


                                      Connected: ip-10-86-222-208.ec2.internal


-= CDS Synchronization Status =-

Last Refreshed: 06:08:28
(updated every 5 seconds, ctrl+c to exit)


cds109 ...................................................... [  UP  ]
cds52 ....................................................... [  UP  ]


Next Sync                    Last Sync                    Last Result         
------------------------------------------------------------------------------
cds109
07-27-2011 10:20             07-27-2011 05:41             Success    

cds52
In Progress                  Never                        Never       


                                      Connected: ip-10-86-222-208.ec2.internal

Comment 8 Sachin Ghai 2011-07-27 10:16:05 UTC

I verified this issue locally as well as on amazon and it is not reproducible with new builds ( pulp 0.211/0.214 and rh-rhui-tools 2.0.41).

Comment 9 wes hayutin 2011-08-01 21:38:38 UTC

moving to release pending

Comment 10 wes hayutin 2012-05-31 12:51:55 UTC

closing out, product released

Note You need to log in before you can comment on or make changes to this bug.