Bug 436381

Created attachment 297104 [details]
Config file

Actually, it's not exactly as described: the quorum forms, but clvmd fails to
set up LVs because a quorum hasn't formed before clvmd comes up, which then
causes GFS volumes to be inaccessible when the 'gfs' init script is started.

We employed a fix on a test cluster which simply involves reversing the start
order of cman and qdiskd.  Because fenced waits up to 5 minutes (by default) for
a quorum to form, qdiskd can actually cause quorum formation while fenced is
waiting.  This of course allows clvmd to start correctly (the first time!),
which, in turn allows gfs file systems to be mounted immediately.

The original reason qdiskd was started after cman was two-fold:
(a) cman was required for qdiskd to run, and 
(b) qdiskd might need clvmd, so it should start after it


In RHEL5, clvmd is started at level 24 while qdiskd is started at 22. 
Therefore, (b) won't occur - and since it is not supported anyway, this is ok.

As of RHEL5.1, qdiskd can start and wait indefinitely for CMAN to come up before
operating in any useful capacity, so the requirement for (a) is negated.

It is my belief that because the cman and qdiskd init scripts are placed in the
same package, we can reverse their ordering safely.

FWIW, cman starts at level 21 and qdiskd at level 22.

Created attachment 297229 [details]
Allow start-if-enabled before fenced in cman script

This patches the cman + qdiskd scripts, making qdiskd start if chkconfig'd on
before fenced.

Failure of qdiskd to start (for example, if it's on clustered LVs) is non-fatal
to the cman-init script, so if chkconfig'd on and moved later in the start tree
(for example after clvmd), it will get started a second time if needed.

Subsequent start-after-start of the qdiskd init script no longer prints "QDisk
services already running", so if the start occurs successfully from the cman
script, the qdiskd script does not misbehave or raise alarms.

Mar  3 17:58:05 ora1 kernel: dlm: no local IP address has been set
Mar  3 17:58:05 ora1 kernel: dlm: cannot start dlm lowcomms -107
Mar  3 17:58:05 ora1 clvmd: Unable to create lockspace for CLVM: Transport

The above is usually due to dlm_controld not being started, or configfs not
being mounted.

I reproduced those messages simply by trying to start clvmd on an inquorate
cluster (after starting cman with the init script, which starts all daemons +
mounts configfs).

* clvmd exits immediately after startup:

  [root@frederick ~]# clvmd -d
  CLVMD[aaee4430]: Mar 31 10:12:43 CLVMD started
  CLVMD[aaee4430]: Mar 31 10:12:43 Connected to CMAN
  CLVMD[aaee4430]: Mar 31 10:12:43 CMAN initialisation complete
  CLVMD[aaee4430]: Mar 31 10:12:43 Can't initialise cluster interface
    Can't initialise cluster interface

  (in dmesg:)
  dlm: no local IP address has been set
  dlm: cannot start dlm lowcomms -107

* if, however, I make the cluster quorate, then inquorate, starting clvmd at
that point produces the following:

  [root@frederick ~]# clvmd -d
  CLVMD[aaee4430]: Mar 31 10:16:13 CLVMD started
  CLVMD[aaee4430]: Mar 31 10:16:13 Connected to CMAN
  CLVMD[aaee4430]: Mar 31 10:16:13 CMAN initialisation complete
  (waits for quorum)

* If at this point, I make the cluster quorate, clvmd wakes up:

  CLVMD[aaee4430]: Mar 31 10:16:55 Cluster ready, doing some more initialisation
  CLVMD[aaee4430]: Mar 31 10:16:55 starting LVM thread
  CLVMD[aaee4430]: Mar 31 10:16:55 clvmd ready for work
  CLVMD[aaee4430]: Mar 31 10:16:55 Using timeout of 60 seconds
  CLVMD[aaee4430]: Mar 31 10:16:55 Got state change message, 
                   re-reading members list
  CLVMD[41401940]: Mar 31 10:16:55 LVM thread function started


So, I don't even think this is an ordering bug.  CLVMD isn't staying up the
first time unless the cluster is quorate.  Mitigating it by flipping the init
script ordering only masks the problem: any time a cluster node boots, if that
cluster node has not been quorate, administrator intervention is required to
bring up clustered LVM volumes after the cluster node becomes a part of the
cluster quorum.

Note that I think this init script patch should be included in the near future,
but ultimately, I do not believe it solves this particular bug.

clvmd is simply the victim in all this. 

The reason it can't start is that the DLM is not being configured by
dlm_controld, either because it hasn't been (fully) started or it has
encountered an error of some kind.

http://sources.redhat.com/git/?p=cluster.git;a=commit;h=c1b276c491b0d6e625035b5063532abc3ce23ca4

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0189.html