Bug 454848

Summary: unstable file system after kernel update
Product: [Fedora] Fedora Reporter: Ralf W. Grosse-Kunstleve <rwgk>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: low    
Version: 8CC: esandeen
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
URL: https://www.redhat.com/archives/fedora-list/2008-July/msg01127.html
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-07-15 00:03:30 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
example output of 99 bin/libtbx.scons runs none

Description Ralf W. Grosse-Kunstleve 2008-07-10 06:49:27 UTC 
Description of problem:

  Randomly, files appear to be missing, leading to application failures.
  When immediately trying again the exact same command it (usually) works.

Version-Release number of selected component (if applicable):

  kernel 2.6.23.15-137.fc8

How reproducible:

  I can reproduce this problem by repeatedly running SCons (make replacement)
  on a large source tree. About 10% of the SCons runs end with some kind
  of problem.
  I have direct evidence that the problem is connected to the kernel version:
  After rebooting with kernel 2.6.25.9-40.fc8 I'm seeing the SCons
  failures.
  After rebooting with the older kernel 2.6.23.15-137.fc8 the machine is
  completely stable again.
  Other machines with a newer kernel are unstable, too.
  Other machines with the older kernel are stable, too.

Steps to Reproduce:
  mkdir /var/tmp/junk
  cd /var/tmp/junk
  wget http://cci.lbl.gov/cctbx_build/results/2008_04_25_1421/cctbx_bundle.selfx
  perl cctbx_bundle.selfx 4 # this will take ~10 minutes
  cd cctbx_build
  bin/libtbx.scons
  bin/libtbx.scons
  bin/libtbx.scons
  ...
  
Actual results:

  About 10% of the time there are SCons warnings or spurious recompilations.

Expected results:

  I'll attach an example output with 99 bin/libtbx.scons results, showing
  8 "missing SConscript" warnings and one spurious set of recompilations.
  The other 90 calls are OK. 

Additional info:

  The same problem exists under Fedora 9, all versions from the day it
  was released (same day I installed it) to at least this kernel version:
  2.6.25.9-76.fc9.x86_64 #1 SMP Fri Jun 27

  The first kernel version I know is broken is 2.6.25.4-10.fc8.

  I've also seen at least one "make" failure, while building
  Python 2.5.2 from sources. So it is not just SCons. SCons
  just happens to trigger the kernel problem exceptionally
  often.
Comment 1 Ralf W. Grosse-Kunstleve 2008-07-10 06:49:27 UTC 
Created attachment 311450 [details]
example output of 99 bin/libtbx.scons runs
Comment 2 Eric Sandeen 2008-07-10 16:31:54 UTC 
i'll take a quick look over the testcase, thanks.
Comment 3 Eric Sandeen 2008-07-10 16:39:05 UTC 
For those playing along at home, you need tcsh to run the testcase.
Comment 4 Eric Sandeen 2008-07-10 19:18:43 UTC 
.... and python-devel.  :)

Anyway, I ran 100 builds on 2.6.25.9-40.fc8 and saw no errors.

On a whim, can you add default_relatime=0 to your boot commandline and see if it
makes any difference?

post-boot grep "default relative atime updates" in dmesg to be sure...

Thanks,
-Eric
Comment 5 Ralf W. Grosse-Kunstleve 2008-07-11 02:02:46 UTC 
> On a whim, can you add default_relatime=0 to your boot commandline and see if it
> makes any difference?

Will do that over the weekend; the machines are too busy right now.

In the meantime, I did this:

1. install fedora 8 from dvd, starting from scratch (i.e. disk repartitioned)
2. yum install tcsh; adduser ralf, no other customizations
3. run 100*bin/libtbx.scons
4. yum update + reboot
5. run 100*bin/libtbx.scons

Step 3. shows no errors.
(I got stuck at step 4 with this: ERROR with rpm_check_debug
but got around this via: yum remove NetworkManager; yum update)
To my surprise, step 5. also shows no errors!
The kernel version after yum update is still as reported yesterday:
2.6.25.9-40.fc8

The only difference to what I did before are the missing customizations
of the system that I usually do: activate NFS (server & client),
NIS, automount, some other misc. things. I'll do that asap and will
report what happens.
Comment 6 Ralf W. Grosse-Kunstleve 2008-07-14 23:44:25 UTC 
Followup to comment #5: I applied all customizations step by step, rebooting
and testing several times. The machine remained stable all the way to the end.

Saturday I ran "yum update" on the Fedora 9 machine, and is is stable now,
too, for the first time.

Today I rebuilt another unstable Fedora 8 system from scratch, exactly the
same way as the one in comment #5, only that I did everything in one go
and didn't reboot+test after each step. At the end is was NOT stable.
I have absolutely no clue how this can be, except that "yum update" may
have given me different updates on Friday (first machine) and today
(second machine). But then again, why should it be "NOT OK", "OK", "NOT OK"?
Nothing here really makes sense.

Rebooting the second FC8 machine with the original kernel (2.6.23.1-42.fc8)
made it stable again. This is similar to what I observed on another machine
mentioned in my original posting.

For completeness, eventually I ran rpm --install --force
kernel-2.6.23.15-137.fc8.x86_64.rpm
kernel-devel-2.6.23.15-137.fc8.x86_64.rpm
kernel-headers-2.6.23.15-137.fc8.x86_64.rpm
on the machine I did today (just because we've been using that kernel
for several months without problems) and it is still stable.

Summary: a few days ago I had three broken machines, today I have them
all fixed somehow in three different ways, and I have zero explanations.

I'll keep my hand off the systems now, quietly hoping that somebody
will somehow find and fix the root cause of the problem, without ever
knowing how much trouble it has caused me.
Comment 7 Eric Sandeen 2008-07-14 23:52:03 UTC 
If you wind up with a problematic system again, please do try turning off the
default relatime and see how that goes.

Thanks,
-Eric
Comment 8 Eric Sandeen 2008-07-15 00:03:30 UTC 
I'm not sure there's a lot we can  do w/o a reproducer but please do keep us up
to date, and re-open, if you get more info.

Thanks,
-Eric