Bug 204500 - memory leak in zlib -1.1.4
memory leak in zlib -1.1.4
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: zlib (Show other bugs)
3.8
All Linux
high Severity medium
: ---
: ---
Assigned To: Ivana Varekova
Jay Turner
:
: 204498 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-08-29 11:33 EDT by Jose Plans
Modified: 2015-01-07 19:14 EST (History)
3 users (show)

See Also:
Fixed In Version: RHBA-2007-0419
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-06-11 14:42:03 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Fix for this bug (500 bytes, patch)
2006-08-29 11:33 EDT, Jose Plans
no flags Details | Diff
Test program attempting to prove the validity of the patch. (4.34 KB, text/x-csrc)
2006-08-29 11:37 EDT, Jose Plans
no flags Details
valgrind output without the patch on the test program. (2.25 KB, application/octet-stream)
2006-08-29 11:39 EDT, Jose Plans
no flags Details
valgrind output with the patch on the test program. (898 bytes, application/octet-stream)
2006-08-29 11:40 EDT, Jose Plans
no flags Details

  None (edit)
Description Jose Plans 2006-08-29 11:33:25 EDT
* Description of problem:

This customer reported a memory leak not always reproducible on zlib 1.1.4
shipped in RHEL3.
They noticed it with their own application, since the problem was not always
reproducible, after identifying the cause of it and contacting upstream, I did
write a test case and the patch to fix it.
This is also not the case on 1.2.0 zlib branch.

Following the notes provided by the maintainer :
This happens in deflate.c by the way.

--- From Issue-Tracker ---
When calling :  int ZEXPORT deflateInit2_ (...)  
|_ We are initialising the structure : deflate_state *s;
|
|_ We allocate a memory are for its pointer :
|  --
|    s = (deflate_state *) ZALLOC(strm, 1, sizeof(deflate_state));
|  --
|  
|_ We allocate other members after it to be handled by s :
|  --
|   s->window = (Bytef *) ZALLOC(strm, s->w_size, 2*sizeof(Byte));
|   s->prev   = (Posf *)  ZALLOC(strm, s->w_size, sizeof(Pos));
|   s->head   = (Posf *)  ZALLOC(strm, s->hash_size, sizeof(Pos));
|
|   s->lit_bufsize = 1 << (memLevel + 6); /* 16K elements by default */
|
|   overlay = (ushf *) ZALLOC(strm, s->lit_bufsize, sizeof(ush)+2);
|  --
|  
|_ Notice now, that overlay will be pointed by :
|  --
|   s->pending_buf = (uchf *) overlay;
|  --
|  If This one failed, then s->pending_buf is Z_NULL.
|  
|_ This leads us now to a test and if one of these members failed because of
|  a memory allocation problem (lack of memory) then we will call deflateEnd().
|  However, in the original code, we never initialised [int status] contained
|  in deflate_state, missing then the memory frees from deflateEnd :
|   
|   --
|   if (s->window == Z_NULL || s->prev == Z_NULL || s->head == Z_NULL ||
|      s->pending_buf == Z_NULL) {
|+     /* initialise the status so we can free at deflateInit2_() */
|+     s->status = FINISH_STATE;
|       strm->msg = (char*)ERR_MSG(Z_MEM_ERROR);
|       deflateEnd (strm);
|       return Z_MEM_ERROR;
|   }
|  --
|  
\_Without initialising s->status, we were calling deflateEnd and :
  
  --
   status = strm->state->status;
   if (status != INIT_STATE && status != BUSY_STATE &&
       status != FINISH_STATE) {
     return Z_STREAM_ERROR;
   }
  --
  
  Here we were checking whether status wasn't INIT_STATE and wasn't BUSY_STATE
  and wasn't FINISH_STATE to return a stream error. However the fact status
  wasn't initialised due to a ZALLOC failure made us entering in that condition.
  With the addition of s->status = FINISH_STATE we can then jump that
  condition and end up freeing allocated memory.
--- From Issue-Tracker ---

I believe the test case attached below is enough for the patch to be proved correct.
However, please let me know if there is something missing.

* Version-Release number of selected component (if applicable):

            zlib-1.1.4-8.1

* How reproducible:

  Only when, for some reasons of memory exhaustion, this ZALLOC call fails:
	
   ---    
     overlay = (ushf *) ZALLOC(strm, s->lit_bufsize, sizeof(ush)+2);
   ---

* Steps to Reproduce:
1. unknown at this point.
  
* Actual results:
---
BackTraceGet[0]=src addr: 0x000002CD host addr 0x001182CD - __builtin_vec_new
BackTraceGet[1]=src addr: 0x001DB0E0 host addr 0x002F30E0 -
CCompressor::zcalloc(void *, unsigned int, unsigned int)
BackTraceGet[2]=src addr: 0x0020B418 host addr 0x00323418 - deflateInit2_
BackTraceGet[3]=src addr: 0x0020B163 host addr 0x00323163 - deflateInit_
BackTraceGet[4]=src addr: 0x0015F591 host addr 0x00277591 -
CAppObjectOnSBP::compressInit(CZlibInfo *)
BackTraceGet[5]=src addr: 0x0015ED7D host addr 0x00276D7D -
CAppObjectOnSBP::DoStreamCompressInit(bool)
BackTraceGet[6]=src addr: 0x001743C0 host addr 0x0028C3C0 -
CHttpApp::FwdStreamingreply(CAppObjectOnSBP *, CAppCxt &)
BackTraceGet[7]=src addr: 0x00178AD1 host addr 0x00290AD1 -
CHttpApp::HandleHttpMessage(CbswSmartPtr<CSpb>, CAppObjectOnSBP *, int)
BackTraceGet[8]=src addr: 0x001786DE host addr 0x002906DE -
CHttpApp::RecvMainLoop(CHttpMsgInternal *)
BackTraceGet[9]=src addr: 0x00179B3A host addr 0x00291B3A -
CHttpApp::TaskRcvPres(int, int, int, int, int, int, int, int, int, int)
--- // ---
    zcalloc()
    deflateInit2_
    deflateInit_
    ...

*or with the test case using a very close approach of the zlib code for deflate*
---
    malloc (vg_replace_malloc.c:149)
    leak_test (leak_deflate.c:59) (same as deflateInit2)
    main (leak_deflate.c:113) (calling deflateInit2 as deflateinit does as parent)
    

Note that both behaviors are the same.

* Expected results:

No leaks.
---
(this is using the trigger program)
==3719== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 12 from 1)
==3719== malloc/free: in use at exit: 0 bytes in 0 blocks.
==3719== malloc/free: 4 allocs, 4 frees, 37 bytes allocated.
==3719== For counts of detected errors, rerun with: -v
==3719== No malloc'd blocks -- no leaks are possible.


* Additional info:
Comment 1 Jose Plans 2006-08-29 11:33:25 EDT
Created attachment 135145 [details]
Fix for this bug
Comment 4 Jose Plans 2006-08-29 11:37:49 EDT
Created attachment 135147 [details]
Test program attempting to prove the validity of the patch.
Comment 5 Jose Plans 2006-08-29 11:39:16 EDT
Created attachment 135148 [details]
valgrind output without the patch on the test program.
Comment 6 Jose Plans 2006-08-29 11:40:18 EDT
Created attachment 135150 [details]
valgrind output with the patch on the test program.
Comment 9 Ivana Varekova 2006-08-31 08:29:05 EDT
*** Bug 204498 has been marked as a duplicate of this bug. ***
Comment 14 Jay Turner 2006-10-30 07:07:08 EST
QE ack for 3.9.
Comment 18 Red Hat Bugzilla 2007-06-11 14:42:03 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2007-0419.html

Note You need to log in before you can comment on or make changes to this bug.