Bug 177024 - Failover to slave lockserver fails after master's network interface brought down
Failover to slave lockserver fails after master's network interface brought down
Status: CLOSED NOTABUG
Product: Red Hat Cluster Suite
Classification: Red Hat
Component: gfs (Show other bugs)
3
i686 Linux
medium Severity high
: ---
: ---
Assigned To: Chris Feist
GFS Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-01-05 09:40 EST by Don Breslauer
Modified: 2010-01-11 22:09 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-03-16 15:33:30 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
/var/log/messages from client (79.12 KB, application/octet-stream)
2006-01-05 09:43 EST, Don Breslauer
no flags Details
/var/log/messages from client 2 (79.07 KB, text/plain)
2006-01-05 09:43 EST, Don Breslauer
no flags Details
/var/log/messages from master lockserver (22.13 KB, text/plain)
2006-01-05 09:44 EST, Don Breslauer
no flags Details
/var/log/messages from slave1 (163.17 KB, text/plain)
2006-01-05 09:44 EST, Don Breslauer
no flags Details
/var/log/messages from slave 2 (174.33 KB, text/plain)
2006-01-05 09:45 EST, Don Breslauer
no flags Details
netstat -nr infralock1 (458 bytes, text/plain)
2006-01-05 10:46 EST, Don Breslauer
no flags Details
cluster.ccs (211 bytes, text/plain)
2006-01-05 10:46 EST, Don Breslauer
no flags Details
fence.ccs (1.42 KB, text/plain)
2006-01-05 10:48 EST, Don Breslauer
no flags Details
node.ccs (1.50 KB, text/plain)
2006-01-05 10:49 EST, Don Breslauer
no flags Details
DR /var/log/messages (1.20 MB, text/plain)
2006-01-05 11:32 EST, Don Breslauer
no flags Details
dr client /var/log/messages snapshot (905 bytes, text/plain)
2006-01-05 11:48 EST, Don Breslauer
no flags Details
dr slave log file (814.85 KB, text/plain)
2006-01-05 12:03 EST, Don Breslauer
no flags Details
result of fence_node <MASTER> command (15.18 KB, text/plain)
2006-01-05 13:26 EST, Don Breslauer
no flags Details
result of fence_node <MASTER> (15.67 KB, text/plain)
2006-01-05 13:26 EST, Don Breslauer
no flags Details
/etc/hosts and ccs archive (22.00 KB, application/octet-stream)
2006-01-06 09:20 EST, Don Breslauer
no flags Details
DR CCS archive (7.00 KB, application/octet-stream)
2006-01-10 17:15 EST, Don Breslauer
no flags Details
Output from commands run on each lockserver (10.00 KB, application/octet-stream)
2006-01-10 21:02 EST, Don Breslauer
no flags Details
/var/log/messages from dr cluster during reboot (125.00 KB, application/octet-stream)
2006-01-11 09:45 EST, Don Breslauer
no flags Details
result of reboot -fin (540.00 KB, application/octet-stream)
2006-01-11 14:46 EST, Don Breslauer
no flags Details

  None (edit)
Description Don Breslauer 2006-01-05 09:40:52 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)

Description of problem:
If we disable eth1 on the lockmaster, the entire cluster gets fenced.
The slaves do not fully arbitrate for Master.
 
All nodes in the cluster have 2 network interfaces. hearbeats and nodes.ccs reference the eth1 interface.
I've attached messages from master/slaves/clients as well as our cca config files.

Version-Release number of selected component (if applicable):
GFS 6.0.2.27

How reproducible:
Always

Steps to Reproduce:
1.disable eth1 network interface on master lock server (gfs heartbeats go over eth1)
2.
3.
  

Actual Results:  /var/log/messages indicates missed heartbeats. cluster then crashes

Expected Results:  slave lockservers should have arbitrated to select new master

Additional info:
Comment 1 Don Breslauer 2006-01-05 09:43:00 EST
Created attachment 122819 [details]
/var/log/messages from client
Comment 2 Don Breslauer 2006-01-05 09:43:33 EST
Created attachment 122820 [details]
/var/log/messages from client 2
Comment 3 Don Breslauer 2006-01-05 09:44:04 EST
Created attachment 122821 [details]
/var/log/messages from master lockserver
Comment 4 Don Breslauer 2006-01-05 09:44:35 EST
Created attachment 122822 [details]
/var/log/messages from slave1
Comment 5 Don Breslauer 2006-01-05 09:45:10 EST
Created attachment 122823 [details]
/var/log/messages from slave 2
Comment 6 Jonathan Earl Brassow 2006-01-05 10:39:26 EST
Please send:

infralock1> netstat -nr

and either the CCS archive or the archive files

Comment 11 Don Breslauer 2006-01-05 10:55:02 EST
As a control test, we changed the ccs files to send gfs heartbeats on eth0
instead of eth1. When we brought down eth0, we saw the same negative result.
Comment 12 Don Breslauer 2006-01-05 10:59:10 EST
We tried the same test in our Disaster Recovery (DR) environment, where there is
only a single network interface. We go the same negative result. 
Comment 13 Jonathan Earl Brassow 2006-01-05 11:11:23 EST
In your DR environment, how does the master fence the other nodes if its only
network interface is down?
Comment 14 Don Breslauer 2006-01-05 11:15:30 EST
In DR, the nodes were not fenced, they just lost GFS.  The cluster fell.  But
the arbitration that we expected to take place between the remaining ("slave")
lockservers did not occur.
Comment 15 Don Breslauer 2006-01-05 11:32:31 EST
Created attachment 122834 [details]
DR /var/log/messages
Comment 16 Jonathan Earl Brassow 2006-01-05 11:37:20 EST
Also, in the controled test (comment 11), none of the nodes got fenced, right? 
The cluster (WRT GFS) just hung?

Can you also get me the log from one of the slave servers in your DR environment?

Comment 17 Don Breslauer 2006-01-05 11:48:31 EST
Created attachment 122836 [details]
dr client /var/log/messages snapshot
Comment 18 Don Breslauer 2006-01-05 12:03:43 EST
Created attachment 122837 [details]
dr slave log file
Comment 19 Jonathan Earl Brassow 2006-01-05 12:14:09 EST
In comment 18, I see that all the fencing actions are failing...
Comment 20 Jonathan Earl Brassow 2006-01-05 12:29:38 EST
Another strange thing I see in comment 18 is that it seems that the slave
(elminfralock2) can't communicate with anyone.  All fencing ops fail and it
can't talk to any of the other nodes... at least that's what it seems like to me.
Comment 21 Don Breslauer 2006-01-05 12:33:39 EST
(In reply to comment #20)
> Another strange thing I see in comment 18 is that it seems that the slave
> (elminfralock2) can't communicate with anyone.  All fencing ops fail and it
> can't talk to any of the other nodes... at least that's what it seems like to me.
> 

We just ran a test - we used fence_node infralock2 from infralock1,where
infralock2 was the master. Everything worked as expected. Successfully
arbirtrated the new master.

Comment 22 Chris Feist 2006-01-05 12:48:56 EST
What command are you using to bring down the interface in your DR environment?
Comment 23 Jonathan Earl Brassow 2006-01-05 12:53:24 EST
Jan  5 11:00:07 elminfralock2 lock_gulmd_LTPX[1400]: New Master at
elminfralock2:10.5.211.49

I see that a new master _is_ being selected, it just can not fence...

can you rerun you fence_node test from infralock2 -> infralock1

We need to be careful that we are keeping our clusters straight.  I see in
comment 18 that elminfralock2 is trying to fence elminfra*, but we keep
referring to 'infra*'.  If the wrong name is being used as an argument to
fence_node, it will fail.
Comment 24 Don Breslauer 2006-01-05 13:04:59 EST
(In reply to comment #23)
> Jan  5 11:00:07 elminfralock2 lock_gulmd_LTPX[1400]: New Master at
> elminfralock2:10.5.211.49
> 
> I see that a new master _is_ being selected, it just can not fence...
> 
> can you rerun you fence_node test from infralock2 -> infralock1
> 
> We need to be careful that we are keeping our clusters straight.  I see in
> comment 18 that elminfralock2 is trying to fence elminfra*, but we keep
> referring to 'infra*'.  If the wrong name is being used as an argument to
> fence_node, it will fail.


We reran the test.  elm* refers to the DR environment. infra* is production.
infralock2: master
infralock2 & infralock3:	slaves

ran the following on slave, infralock1:
[root@infralock1 sbin]# /sbin/fence_node infralock2
Performing fence method, riloe, on infralock2.
Warning:  There are no node specific fencing parameters.
The agent (fence_ilo) reports: success

Fence of "infralock2" was successful.

The lockmaster fenced correctly, the slaves arbitraded and a new master was
delegated:


messages on slave during fence:
Jan  5 12:34:01 infralock1 lock_gulmd_core[4079]: Failed to receive a timely
heartbeat repl
y from Master. (t:1136482441999893 mb:1)
Jan  5 12:34:06 infralock1 lock_gulmd_core[4079]: Failed to receive a timely
heartbeat repl
y from Master. (t:1136482446999968 mb:2)
Jan  5 12:34:12 infralock1 lock_gulmd_core[4079]: Failed to receive a timely
heartbeat repl
y from Master. (t:1136482452000045 mb:3)
Jan  5 12:34:18 infralock1 xinetd[4407]: START: echo-dgram pid=0 from=208.34.187.94
Jan  5 12:34:28 infralock1 lock_gulmd_core[4079]: Timeout (15000000) on fd:5
(infralock2:10
.100.202.60)
Jan  5 12:34:28 infralock1 lock_gulmd_core[4079]: Node infraqa1 dissapeared
while we were w
ithout a Master.
Jan  5 12:34:28 infralock1 lock_gulmd_core[4079]: Found Master at infralock3, so
I'm a Slav
e.
Jan  5 12:34:28 infralock1 lock_gulmd_core[4079]: Failed to receive a timely
heartbeat repl
y from Master. (t:1136482468040760 mb:1)
Jan  5 12:34:28 infralock1 lock_gulmd_LT000[4082]: New Master at
infralock3:10.100.202.61
Jan  5 12:34:28 infralock1 lock_gulmd_LTPX[4085]: New Master at
infralock3:10.100.202.61
Jan  5 12:34:28 infralock1 lock_gulmd_LTPX[4085]: Logged into LT000 at
infralock3:10.100.20
2.61
Jan  5 12:34:28 infralock1 lock_gulmd_LTPX[4085]: Finished resending to LT000
Jan  5 12:34:28 infralock1 lock_gulmd_LT000[4082]: Logged into Master at
infralock3:10.100.
202.61
Jan  5 12:34:44 infralock1 fence_node[11299]: The agent (fence_ilo) reports: success
Jan  5 12:34:44 infralock1 fence_node[11299]: Fence of "infralock2" was successful.



messages on slave:

Jan  5 12:33:59 infralock3 lock_gulmd_core[5506]: Failed to receive a timely
heartbeat repl
y from Master. (t:1136482439645984 mb:1)
Jan  5 12:34:04 infralock3 lock_gulmd_core[5506]: Failed to receive a timely
heartbeat repl
y from Master. (t:1136482444646181 mb:2)
Jan  5 12:34:09 infralock3 lock_gulmd_core[5506]: Failed to receive a timely
heartbeat repl
y from Master. (t:1136482449646382 mb:3)
Jan  5 12:34:19 infralock3 xinetd[4399]: START: echo-dgram pid=0 from=208.34.187.94
Jan  5 12:34:25 infralock3 lock_gulmd_core[5506]: Timeout (15000000) on fd:5
(infralock2:10
.100.202.60)
Jan  5 12:34:25 infralock3 lock_gulmd_core[5506]: I see no Masters, So I am
Arbitrating unt
il enough Slaves talk to me.
Jan  5 12:34:25 infralock3 lock_gulmd_core[5506]: Could not send quorum update
to slave inf
ralock1
Jan  5 12:34:25 infralock3 lock_gulmd_core[5506]: Could not send quorum update
to slave inf
raqa1
Jan  5 12:34:25 infralock3 lock_gulmd_core[5506]: Could not send quorum update
to slave inf
ralock3
Jan  5 12:34:25 infralock3 lock_gulmd_core[5506]: LastMaster
infralock2:10.100.202.60, is b
eing marked Expired.
Jan  5 12:34:25 infralock3 lock_gulmd_core[5506]: Could not send membership
update "Expired
" about infralock2 to slave infralock1
Jan  5 12:34:25 infralock3 lock_gulmd_core[5506]: Could not send membership
update "Expired
" about infralock2 to slave infraqa1
Jan  5 12:34:25 infralock3 lock_gulmd_LTPX[5512]: New Master at
infralock3:10.100.202.61
Jan  5 12:34:25 infralock3 lock_gulmd_core[5506]: Could not send membership
update "Expired
" about infralock2 to slave infralock3
Jan  5 12:34:25 infralock3 lock_gulmd_core[11108]: Gonna exec fence_node infralock2
Jan  5 12:34:25 infralock3 lock_gulmd_core[5506]: Forked [11108] fence_node
infralock2 with
 a 0 pause.
Jan  5 12:34:25 infralock3 fence_node[11108]: Performing fence method, riloe, on
infralock2
.
Jan  5 12:34:25 infralock3 fence_node[11108]: Warning:  There are no node
specific fencing
parameters.
Jan  5 12:34:26 infralock3 lock_gulmd_core[5506]: Still in Arbitrating: Have 1,
need 2 for
quorum.
Jan  5 12:34:28 infralock3 lock_gulmd_core[5506]: Now have Slave quorum, going
full Master.

Jan  5 12:34:28 infralock3 lock_gulmd_core[5506]: New Client: idx:1 fd:5 from
(10.100.202.5
9:infralock1)
Jan  5 12:34:28 infralock3 lock_gulmd_LTPX[5512]: Logged into LT000 at
infralock3:10.100.20
2.61
Jan  5 12:34:28 infralock3 lock_gulmd_LT000[5509]: New Client: idx 2 fd 7 from
(10.100.202.
61:infralock3)
Jan  5 12:34:28 infralock3 lock_gulmd_LTPX[5512]: Finished resending to LT000
Jan  5 12:34:28 infralock3 lock_gulmd_LT000[5509]: Attached slave
infralock1:10.100.202.59
idx:3 fd:8 (soff:3 connected:0x8)
Jan  5 12:34:28 infralock3 lock_gulmd_LT000[5509]: New Client: idx 4 fd 9 from
(10.100.202.
59:infralock1)
Jan  5 12:34:43 infralock3 lock_gulmd_core[5506]: New Client: idx:4 fd:9 from
(10.100.202.1
:infraqa1)
Jan  5 12:34:43 infralock3 lock_gulmd_LT000[5509]: New Client: idx 5 fd 10 from
(10.100.202
.1:infraqa1)
Jan  5 12:35:25 infralock3 fence_node[11108]: The agent (fence_ilo) reports: success
Jan  5 12:35:25 infralock3 fence_node[11108]: Fence of "infralock2" was successful.
Jan  5 12:35:25 infralock3 lock_gulmd_core[5506]: found match on pid 11108,
marking node in
fralock2 as logged out.
Jan  5 12:37:23 infralock3 lock_gulmd_core[5506]: New Client: idx:5 fd:10 from
(10.100.202.
2:infraqa2)
Jan  5 12:37:23 infralock3 lock_gulmd_LT000[5509]: New Client: idx 6 fd 11 from
(10.100.202
.2:infraqa2)
Jan  5 12:37:56 infralock3 xinetd[4399]: START: echo-dgram pid=0 from=208.34.187.94
Jan  5 12:38:07 infralock3 lock_gulmd_core[5506]: Generation ID of
client(infralock2) (1136
482651671549) is not the same as ours (1136409415394275)
Jan  5 12:38:10 infralock3 lock_gulmd_core[5506]: New Client: idx:6 fd:11 from
(10.100.202.
60:infralock2)
Jan  5 12:38:10 infralock3 lock_gulmd_LT000[5509]: New Client: idx 7 fd 12 from
(10.100.202
.60:infralock2)
Jan  5 12:38:10 infralock3 lock_gulmd_LT000[5509]: Attached slave
infralock2:10.100.202.60
idx:8 fd:13 (soff:2 connected:0xc)
Comment 25 Don Breslauer 2006-01-05 13:06:08 EST
(In reply to comment #22)
> What command are you using to bring down the interface in your DR environment?

ifdown eth1
Comment 26 Don Breslauer 2006-01-05 13:25:10 EST
We ran the fence_node command another time, and got a different unexpected
result. the master was fenced--correct behavior-- but so were some of the
nodes--incorrect behavior. I'm attaching the log files.
Comment 27 Don Breslauer 2006-01-05 13:26:06 EST
Created attachment 122839 [details]
result of fence_node <MASTER> command
Comment 28 Don Breslauer 2006-01-05 13:26:44 EST
Created attachment 122840 [details]
result of fence_node <MASTER>
Comment 29 Chris Feist 2006-01-05 18:00:56 EST
I've got a couple more questions for you.  What is the version of GFS that
you're running on all of the nodes?  Can you also tar up all the /etc/hosts
files on all of the nodes and post them here?

It looks like we've got two seperate problems to solve.  Why is the fence
failing on the slaves after the master has been removed, and why are multiple
nodes being fenced when the master is killed?  I'll continue examining the logs
to see if I can find a clue.
Comment 30 Don Breslauer 2006-01-06 09:19:35 EST
(In reply to comment #29)
> I've got a couple more questions for you.  What is the version of GFS that
> you're running on all of the nodes?  Can you also tar up all the /etc/hosts
> files on all of the nodes and post them here?
> 
> It looks like we've got two seperate problems to solve.  Why is the fence
> failing on the slaves after the master has been removed, and why are multiple
> nodes being fenced when the master is killed?  I'll continue examining the logs
> to see if I can find a clue.


GFS 6.0
RH ES3 Update 2

5 gfs nodes
2 slaves
1 lockmaster

The /etc/hosts on all gfs hosts are idential.
I've attached the ccs archive also.

Comment 31 Don Breslauer 2006-01-06 09:20:38 EST
Created attachment 122872 [details]
/etc/hosts and ccs archive
Comment 32 Chris Feist 2006-01-06 11:56:32 EST
Thanks, can you also give me the output of 'rpm -q GFS' & uname -a.

I just need to verify that I've got the exact same packages to replicate this in
our lab.
Comment 33 Don Breslauer 2006-01-06 12:00:24 EST
(In reply to comment #32)
> Thanks, can you also give me the output of 'rpm -q GFS' & uname -a.
> 
> I just need to verify that I've got the exact same packages to replicate this in
> our lab.

GFS-6.0.2.27-1.debug_statfs.14
Linux infralock1 2.4.21-32.0.1.ELsmp #1 SMP Tue May 17 17:52:23 EDT 2005 i686
i686 i386 GNU/Linux
Comment 34 Chris Feist 2006-01-10 16:53:42 EST
Can you provide the /etc/hosts and ccs files from the DR environment.  It
appears that the other slave lock server loses all network connectivity at the
same time the master lock server loses connectivity.  Is the DNS server running
on the main lock server?
Comment 35 Don Breslauer 2006-01-10 17:12:32 EST
(In reply to comment #34)
> Can you provide the /etc/hosts and ccs files from the DR environment.  It
> appears that the other slave lock server loses all network connectivity at the
> same time the master lock server loses connectivity.  Is the DNS server running
> on the main lock server?

We run NIS in DR.
There are no local hosts entries except for the following on all the gfs nodes &
servers:

127.0.0.1         localhost.localdomain localhost
10.5.210.40     elmnis1
10.5.210.41     elmnis2 loghost

I am attaching the CCS archives for DR
[root@elminfra1 root]#
[root@elminfra1 root]# nslookup elminfralock1
Note:  nslookup is deprecated and may be removed from future releases.
Consider using the `dig' or `host' programs instead.  Run nslookup with
the `-sil[ent]' option to prevent this message from appearing.
Server:         192.104.153.6
Address:        192.104.153.6#53

Name:   elminfralock1.mlp.com
Address: 10.5.211.48
Comment 36 Don Breslauer 2006-01-10 17:15:24 EST
Created attachment 123018 [details]
DR CCS archive
Comment 37 Chris Feist 2006-01-10 17:26:16 EST
Can you provide the output of the following from all 3 lock servers (in your DR
cluster).

host elminfralock1
host elminfralock2
host elminfralock3

ypmatch elminfralock1 hosts
ypmatch elminfralock2 hosts
ypmatch elminfralock3 hosts

ping -c 1 elminfralock1
ping -c 1 elminfralock2
ping -c 1 elminfralock3

Also, can you try the following in your DR cluster:
Make sure all nodes are up and working and restart the master.  Do the slaves
figure out who should be master, and does the old master come back up and become
a slave?
Comment 38 Don Breslauer 2006-01-10 21:01:21 EST
(In reply to comment #37)
> Can you provide the output of the following from all 3 lock servers (in your 
DR
> cluster).
> host elminfralock1
> host elminfralock2
> host elminfralock3
> ypmatch elminfralock1 hosts
> ypmatch elminfralock2 hosts
> ypmatch elminfralock3 hosts
> ping -c 1 elminfralock1
> ping -c 1 elminfralock2
> ping -c 1 elminfralock3
> Also, can you try the following in your DR cluster:
> Make sure all nodes are up and working and restart the master.  Do the slaves
> figure out who should be master, and does the old master come back up and 
become
> a slave?

I am attaching the output.  I will do this test tomorrow morning.

Comment 39 Don Breslauer 2006-01-10 21:02:23 EST
Created attachment 123027 [details]
Output from commands run on each lockserver
Comment 40 Don Breslauer 2006-01-11 09:44:42 EST
(In reply to comment #37)
> Can you provide the output of the following from all 3 lock servers (in your DR
> cluster).
> 
> host elminfralock1
> host elminfralock2
> host elminfralock3
> 
> ypmatch elminfralock1 hosts
> ypmatch elminfralock2 hosts
> ypmatch elminfralock3 hosts
> 
> ping -c 1 elminfralock1
> ping -c 1 elminfralock2
> ping -c 1 elminfralock3
> 
> Also, can you try the following in your DR cluster:
> Make sure all nodes are up and working and restart the master.  Do the slaves
> figure out who should be master, and does the old master come back up and become
> a slave?

The reboot in DR looks good. I'm attaching the /var/log/messages.
Comment 41 Don Breslauer 2006-01-11 09:45:37 EST
Created attachment 123053 [details]
/var/log/messages from dr cluster during reboot
Comment 42 Chris Feist 2006-01-11 12:09:02 EST
Thanks, I've looked through the logs and everything looks good.  Now can you run
the following command on the master 'reboot -fin' and send me the logs from the
lock servers from when you run that command until the master comes back up and
rejoins the cluster?

This should cause the master to be fenced and will let me make sure the
appropriate fence commands are run.
Comment 43 Don Breslauer 2006-01-11 14:44:27 EST
(In reply to comment #42)
> Thanks, I've looked through the logs and everything looks good.  Now can you 
run
> the following command on the master 'reboot -fin' and send me the logs from 
the
> lock servers from when you run that command until the master comes back up 
and
> rejoins the cluster?
> This should cause the master to be fenced and will let me make sure the
> appropriate fence commands are run.
 
when i issue "reboot -fin" on elminfralock1, master lockserver.
 
the 2 slave are in Arbitraiting & Pending state. then slave elminfralock3 
(Pending) gets fenced
the gfs clients hang on gfs volume, all clients except elminfra4 get fenced
 
Attached are all the log files.
 


Comment 44 Don Breslauer 2006-01-11 14:46:04 EST
Created attachment 123065 [details]
result of reboot -fin
Comment 45 Chris Feist 2006-03-16 15:33:30 EST
Moving this bug to NOTABUG.  Currently it's not possible to run heartbeating and
fencing actions on different networks.  If any of the other issues have not been
solved please either re-open this bug or contact me directly.

Note You need to log in before you can comment on or make changes to this bug.