Bug 489658 - autofs crashes because of double-free under heavy stress
Summary: autofs crashes because of double-free under heavy stress
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: autofs
Version: 5.3
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Ian Kent
QA Contact: BaseOS QE
URL:
Whiteboard:
Depends On:
Blocks: 539667
TreeView+ depends on / blocked
 
Reported: 2009-03-11 09:13 UTC by Olivier Fourdan
Modified: 2018-10-20 03:19 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, the autofs code contained a logic error that resulted in a crash under conditions of heavy load. When autofs was not able to create a new pthread, it would double free a value. Now, with the error corrected, when heavily loaded, autofs will fail to create a new pthread safely. It reports the failure, but does not crash.
Clone Of:
: 539667 (view as bug list)
Environment:
Last Closed: 2009-09-02 11:59:23 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Proposed patch (463 bytes, patch)
2009-03-11 09:13 UTC, Olivier Fourdan
no flags Details | Diff
Upstream patch for double free bug (708 bytes, patch)
2009-03-11 14:41 UTC, Ian Kent
no flags Details | Diff
Fix typo in double free patch (707 bytes, patch)
2009-03-11 14:43 UTC, Ian Kent
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2009:1397 0 normal SHIPPED_LIVE autofs bug fix update 2009-09-01 12:02:15 UTC

Description Olivier Fourdan 2009-03-11 09:13:04 UTC
Created attachment 334764 [details]
Proposed patch

Description of problem:

Customer has run heavy stress tests on autofs and discovered a case of double free on an error path.

Version-Release number of selected component (if applicable):

autofs-5.0.1-0.rc2.88.ia64.rpm

How reproducible:

Happened only once, requires stressing autofs until pthread_create() fails to trigger the bug in the error path.

Steps to Reproduce:
  
Actual results:

automount fails with error:

automount[5611]: expire_proc:  expire thread create for /net failed
and automount aborts.

Expected results:

automount fails but does not abort.

Additional info:

Backtrace of the automount process:

#0  0xa000000000010620 in __kernel_syscall_via_break ()
#1  0x20000008001c88e0 in raise () from /lib/libc.so.6.1
#2  0x20000008001cbc60 in abort () from /lib/libc.so.6.1
#3  0x200000080023b3d0 in __libc_message () from /lib/libc.so.6.1
#4  0x200000080024d8b0 in _int_free () from /lib/libc.so.6.1
#5  0x200000080024e1f0 in free () from /lib/libc.so.6.1
#6  0x2000000800028d90 in expire_proc (ap=0x200000080041fc40, now=0) at state.c:305
#7  0x2000000800029c80 in run_state_task (task=<value optimized out>) at state.c:636
#8  0x200000080002c460 in st_queue_handler (arg=0x200000080041a4c0) at state.c:985
#9  0x20000008000f34a0 in start_thread () from /lib/libpthread.so.0
#10 0x2000000800326f60 in __clone2 () from /lib/libc.so.6.1

That comes from state.c

      status = pthread_create(&thid, &thread_attr, expire, ea);
      if (status) {
              error(ap->logopt,
                    "expire thread create for %s failed", ap->path);
              expire_proc_cleanup((void *) ea);
              free(ea);
              return EXP_ERROR;
      }

expire_proc_cleanup() frees the passed value, and it gets freed again just after.

I checked upstream code and it's the same, so this should be fixed upstream similarly.

Comment 1 Ian Kent 2009-03-11 14:30:55 UTC
(In reply to comment #0)
> Created an attachment (id=334764) [details]
> Proposed patch
> 
> Description of problem:
> 
> Customer has run heavy stress tests on autofs and discovered a case of double
> free on an error path.

Yep, good catch, thanks.

snip ...

> That comes from state.c
> 
>       status = pthread_create(&thid, &thread_attr, expire, ea);
>       if (status) {
>               error(ap->logopt,
>                     "expire thread create for %s failed", ap->path);
>               expire_proc_cleanup((void *) ea);
>               free(ea);
>               return EXP_ERROR;
>       }
> 
> expire_proc_cleanup() frees the passed value, and it gets freed again just
> after.

Clearly a bug.

> 
> I checked upstream code and it's the same, so this should be fixed upstream
> similarly.  

Absolutly, I will commit this upstream as well.

Ian

Comment 2 Ian Kent 2009-03-11 14:41:30 UTC
Created attachment 334805 [details]
Upstream patch for double free bug

I can't see anything else that needs doing so this is likley
what will be committed. This should also apply to our RHEL-5
autofs without any trouble.

Comment 3 Ian Kent 2009-03-11 14:43:15 UTC
Created attachment 334806 [details]
Fix typo in double free patch

Comment 5 Ian Kent 2009-03-12 03:33:34 UTC
I have committed this change and built package
autofs-5.0.1-0.rc2.109.src.rpm.

Can we make this available for testing please.

Comment 7 Ian Kent 2009-03-12 09:29:22 UTC
It doesn't matter.

Rev 110 just removes a extra "error(...)" for debugging that
I mistakenly left in a previous patch. I updated the patch
now because I have other patches I'm applying for 5.4 that
change that bit of code also.

Comment 9 Ian Kent 2009-05-21 03:54:49 UTC
This issue has been fixed in the latest autofs package
autofs-5.0.1-0.rc2.125.

I've not been able to duplicate this issue and we haven't seen
it hasn't been reported elsewhere. But, after it being pointed
out it is quite obvious, is a straight forward change and has
been verified to function as expected by the customer.

Comment 11 Chris Ward 2009-07-03 18:26:56 UTC
~~ Attention - RHEL 5.4 Beta Released! ~~

RHEL 5.4 Beta has been released! There should be a fix present in the Beta release that addresses this particular request. Please test and report back results here, at your earliest convenience. RHEL 5.4 General Availability release is just around the corner!

If you encounter any issues while testing Beta, please describe the issues you have encountered and set the bug into NEED_INFO. If you encounter new issues, please clone this bug to open a new issue and request it be reviewed for inclusion in RHEL 5.4 or a later update, if it is not of urgent severity.

Please do not flip the bug status to VERIFIED. Only post your verification results, and if available, update Verified field with the appropriate value.

Questions can be posted to this bug or your customer or partner representative.

Comment 12 Karel Volný 2009-07-29 12:35:50 UTC
SanityOnly - there is the patch mentioned within the bugreport:
> Patch190: autofs-5.0.1-fix-double-free-in-expire_proc.patch
and it applies cleanly during the build

Comment 14 Ruediger Landmann 2009-08-31 05:29:33 UTC
Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

New Contents:
Previously, the autofs code contained a logic error that resulted in a crash under conditions of heavy load. When autofs was not able to create a new pthread, it would double free a value. Now, with the error corrected, when heavily loaded, autofs will fail to create a new pthread safely. It reports the failure, but does not crash.

Comment 15 errata-xmlrpc 2009-09-02 11:59:23 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1397.html


Note You need to log in before you can comment on or make changes to this bug.