Bug 63041 - Quorumd starvation caused shootdown
Quorumd starvation caused shootdown
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 2.1
Classification: Red Hat
Component: clumanager (Show other bugs)
2.1
All Linux
medium Severity medium
: ---
: ---
Assigned To: Lon Hohberger
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2002-04-09 11:42 EDT by Tim Burke
Modified: 2008-05-01 11:38 EDT (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2002-06-11 15:39:41 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Tim Burke 2002-04-09 11:42:46 EDT
From Bugzilla Helper:
User-Agent: Mozilla/4.78 [en] (X11; U; Linux 2.4.7-10 i686)

Description of problem:
clumanager 1.0.9

Tom established a scenario where a cluster shootdown occurred due to multiple
concurrent IO exercisers causing cluquorumd to not be scheduled to run.

Several potential remidies were proposed, including:
- quorumd's nice level was 1 lower than the other daemons.
- scale the cluster lock timeout dymanically based on the tunable parameters
govering the quorumd algorithm to declare a node down.
- breakup a svcmgr loop over 99 services, so it doesn't hold the lock too long.


Version-Release number of selected component (if applicable):


How reproducible:
Sometimes

Steps to Reproduce:
1.Heavy load of multiple independent disk exercisers
2.Enabling debug level logging seemed to exacerbate the problem.
3.
	

Actual Results:  shootdown

Expected Results:  Quorumd should have been scheduled to run.

Additional info:
Comment 1 Tim Burke 2002-05-01 10:29:35 EDT
I also got a similar scenario of quorumd starvation as follows:

An external FC raid array configured as a single disk.  This was partitioned up
into 16 slices of /dev/sdb.  The power switch type was set to watchdog.  On each
cluster member the following was running:
- the cluster software itself
- 5 separate dd processes to different partitions (each repeatedly run out of a
while loop script)
- clustat -i 1
- a script in a while loop would run `cat /proc/slabinfo` once a second

After about a half an hour each system just rebooted (not at the same time,
about 15 minutes appart).  Larry and I looked for traces of an oops, but none
could be found (in /var/log/messages upon reboot).  Therefore, we concluded that
the watchdog timer was exploding after the 10 second interval had elapsed.
Comment 2 Lon Hohberger 2002-06-11 15:39:35 EDT
Fixes in pool, awaiting testing by other developers.

Note You need to log in before you can comment on or make changes to this bug.