Bug 161808 - nodes don't wait for first mounter to finish recovery
nodes don't wait for first mounter to finish recovery
Product: Red Hat Cluster Suite
Classification: Red Hat
Component: gfs (Show other bugs)
All Linux
medium Severity medium
: ---
: ---
Assigned To: David Teigland
GFS Bugs
Depends On:
  Show dependency treegraph
Reported: 2005-06-27 11:15 EDT by David Teigland
Modified: 2010-01-11 22:05 EST (History)
0 users

See Also:
Fixed In Version: RHBA-2005-740
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2005-10-07 12:56:51 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description David Teigland 2005-06-27 11:15:58 EDT
Description of problem:
When the first node mounts a gfs fs, other nodes can't be allowed
to mount until that first node has recovered all journals and
called others_may_mount().  It's the job of lock_dlm to do this,
but it's not -- it allows other nodes to mount while the first
may still be recovering journals.

This can potentially lead to fs corruption.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. all mounted nodes fail at once and are reset
2. all nodes come back and mount at once 
3. the first mounter doesn't complete recovery
of all journals before others also mount
Actual results:

Expected results:

Additional info:
Comment 1 David Teigland 2005-06-27 11:19:23 EDT
Email from Ken:

there is a chance of corruption.  The scenario I see is:

1) 3 machines are mounted.
2) They all fail at the same time.
3) Machine A comes back up and starts replay on the three journals
4) Machine B comes back up, replays it's own journal really quickly
   while Machine A is still working on the first journal.
5) Machine B starts a workload and comes across blocks that are inconsistent
   because the third journal hasn't been replayed yet.  Because all the
   machines died, there are not expired locks to protect the data.

In order to hit the failure case, you always need at least three nodes
to have been mounted at one time or another.  But not all three nodes
need to be running at the power failure time.  (They key is that there
must be a dirty journal beyond the first two to be mounted.)
Comment 2 David Teigland 2005-06-29 03:32:34 EDT
Fixed on RHEL4 and STABLE branches.

The likelihood of this bug causing a problem or corruption is
even smaller than originally thought.  Even if the lock module
doesn't prevent other mounts until first recovery is done,
there's a gfs lock the other mounters block on that has nearly
the same effect.
Comment 4 Red Hat Bugzilla 2005-10-07 12:56:51 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.