Description of problem:

Multi-threaded customer application, which invokes popen(3) internally,
can cause thread stall by FILE lock corruption. Custom application stalled two times this month during operation. The application program is waiting for FILE lock in glibc, but the lock seems to be corrupted. 

Version-Release number of selected component (if applicable):

2.6.18-194

How reproducible:

Once every few weeks of real-world usage.  The custom reproducer (to be attached) should happen once every 10 minutes.

Steps to Reproduce:

This is simplified real-app reproducer.

16 threads do popen(3) invoking gunzip and read gunzip-ped data through
pipe using fread(3). Main thread is watching if other 16 threads are
running.

You can run it:
----------------------------------------------------------------
# tar xvzf popen-reproducer.tgz
# sh MK
# ./pt-popen
---------------------------------------------------------------- 

Actual results:

After start running, you will see "thr XX stalled" assertion report.

----------------------------------------------------------------
# ./pt_popen
thr 12 stalled, fp=0x404ba50, futex=2, nest=-1, owner=4288e940
----------------------------------------------------------------

The victim thread stalls in fread(3) and "fp" is the FILE pointer.
The "futex", "nest" and owner is FILE->_lock contents.
Please notice FILE->_lock.nest should be >=0 and is corrupted.
For detailed scenario, please refer to Additonal Info:

Expected results:

The program run forever without assertion report.

Business impact:

This issue was detected on customer's production application.
And it is obviously considered a data corruption:


Additional info:


* Upstream patch
commit 945754a1754f9d4c2974a8241ad4f92fad7f3a6a

* Stall scenario details between popen(3) and fread(3)

popen(3) creates child process and both parent and child write fp->pid causing do_wp_page() race. And then, child scans all FILE and initializes (zero-ing) fp->_lock.

Here, the other thread doing fread(3) will try to unlock the fp, it can
peek the zeroed nest value and it makes nest value to -1.
The unlock code set owner to 0 if and only if the nest value becomes 0
and owner is left 'owned' state.

Then victim thread accesses the fp, it will lock it and wait for wake-up
because owner is left 'owned', but nobody wakes him up.

* More simple reproducer

Please find the attached corruption-reproduce.tgz to demonstrate
peeking the child process memory.

One thread repeatedly invokes fork(2) and both parent and child write
the data region of same virtual address and break COW.

Parent writes 0xffffffff, child writes 0xa5a5a5a5.

Another thread in parent is watching parent data region and it checks
if the data value is correctly parent side value (0xffffffff)

You can run it:
----------------------------------------------------------------
# tar xvzf corruption-reproduce.tgz.
# sh MK
# ./pt-fork-simple2
----------------------------------------------------------------

You will see data corruption assersion:
----------------------------------------------------------------
# ./pt-fork-simple2
!!! corrupt !!! read=a5a5a5a5
----------------------------------------------------------------