Description of problem:

Nate hit this problem while doing clvm tests on buzz 01-05.

[root@buzz-01 ~]# dlm_tool lockdebug clvmd
Resource len 64  "3jjUnzbhMGE8eVRtuIt2uenthHP6MUIH6PgFiLBO0Z7JP7yuzRRx2VLKvOe8w3rj"
Local 3
Granted
012f0001 CR      Master:   3 03f60001


[root@buzz-02 ~]# dlm_tool lockdebug clvmd
Resource len 64  "3jjUnzbhMGE8eVRtuIt2uenthHP6MUIH6PgFiLBO0Z7JP7yuzRRx2VLKvOe8w3rj"
Local 3
Granted
02cc0002 CR      Master:   3 01d00001

[root@buzz-03 ~]# dlm_tool lockdebug clvmd
Resource len 64  "3jjUnzbhMGE8eVRtuIt2uenthHP6MUIH6PgFiLBO0Z7JP7yuzRRx2VLKvOe8w3rj"
Master
Granted
03f60001 CR      Remote:   1 012f0001
01d00001 CR      Remote:   2 02cc0002
00940001 CR      Remote:   5 038f0001
01850001 CR      Remote:   4 034f0001
01330001 CR

[root@buzz-04 ~]# dlm_tool lockdebug clvmd
Resource len 64  "3jjUnzbhMGE8eVRtuIt2uenthHP6MUIH6PgFiLBO0Z7JP7yuzRRx2VLKvOe8w3rj"
Local 3
Granted
034f0001 CR      Master:   3 01850001

[root@buzz-05 ~]# dlm_tool lockdebug clvmd
Resource len  9  "P_#global"
Master
Granted
01cb0001 PR

Resource len 64  "3jjUnzbhMGE8eVRtuIt2uenthHP6MUIH6PgFiLBO0Z7JP7yuzRRx2VLKvOe8w3rj"
Local 3
Granted
038f0001 CR      Master:   3 00940001

Resource len 18  "V_stripe_8_4096_16"
Local 1          flags 00000000 first_lkid 1360003 root 0 recover 0 locks 0

Expecting reply
nodeid  1 msg request lkid 01360003 resource "V_stripe_8_4096_16 D;"

dmesg from buzz-05
dlm: clvmd: total members 5 error 0
dlm: clvmd: dlm_recover_directory
dlm: clvmd: dlm_recover_directory 0 entries
dlm: clvmd: recover 5 done: 147 ms
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply 1360003 1 master diff 1 -53
dlm: clvmd: receive_request_reply from 1 no lkb 1af0001

It appears that "1af0001" from nodeid 1 should have been "1360003".
On nodeid 1, receive_request() was probably returning another -53
error using setup_stub_lkb().  Because the stub_lkb is shared
for the lockspace among all threads, I suspect the lkid value
is being clobbered by multiple threads using the stub_lkb (a similar
problem with the stub_ms was fixed upstream.)  Ultimately, the
stub_lkb should either have a lock protecting it, or be eliminated.
But, none of this would happen unless there were multiple dlm_recv
threads using it (either concurrently, or maybe due to one sleeping in lowcomms_get_buffer.)

At some point in the distant past, dlm_recv was incorrectly set as
a multithreaded workqueue, but the dlm has never really been made
to work correctly with mutiple recv threads (the stub issues above
being one of the reasons.)  For some reason, these multiple threads
apparently never ran concurrently, so they never posed a problem,
so we never fixed dlm_recv to be single threaded as it should be.

Eventually upstream, the wq threads began operating concurrently,
these problems started appearing, so we fixed it to be single
threaded.  Making a similar fix in rhel would probably resolve
the problem above.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info: