Bug 474210
| Summary: | oom killer implicates dlm during clvmd startup hang | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Nate Straz <nstraz> |
| Component: | lvm2-cluster | Assignee: | LVM and device-mapper development team <lvm-team> |
| Status: | CLOSED DUPLICATE | QA Contact: | Cluster QE <mspqa-list> |
| Severity: | medium | Docs Contact: | |
| Priority: | low | ||
| Version: | 5.3 | CC: | agk, building, ccaulfie, dwysocha, edamato, fj5075fv, heinzm, jbrassow, mbroz, prockai, teigland |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2009-07-14 19:32:34 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Nate Straz
2008-12-02 19:58:31 UTC
It would be good to have dlm debugging messages from all the nodes when this happens. Add <dlm log_debug="1"/> to cluster.conf and we should see them on the console and /var/log/messages. I've been running smack into this while trying to put together a small cluster to demo to the boss. It's not entirely reproducible, but after tearing down and rebuilding the whole thing a few times I've worked out some common threads.
It seems to only affect the last (numerically last) node, and some communications failure with node 1 always seems to be involved. When running in two-node mode, node 2 would be the one to have problems. After adding a third node, node 3 always seems to be the one where dlm_send freaks out. Concurrent with that are the following odd occurrences:
[root@node3 ~]# clustat
Cluster Status for test-cluster @ Tue Jan 13 11:55:55 2009
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
node1-cluster 1 Offline
node2-cluster 2 Online
node3-cluster 3 Online, Local
...this is inconsistent with node1 and node2, who both report everything as kosher with all members and services (I also note that node3 hasn't gotten as far as firing up rgmanager).
There's also this bizarre log entry:
Jan 13 11:46:15 node3 openais[5238]: [CMAN ] Node 1 conflict, remote cluster name='test-clusternode1-bios', local='test-cluster'
...where "node1-bios" refers to the DRAC fence device for node1. I've seen this each time I've run into the dlm_send problem, whether they're related or not I couldn't tell you. I can say that the most recent iteration of the problem did not crop up until I configured fencing for the nodes (note, the fencing devices were configured but unused, and no problems). It all smacks of some strange configuration parsing problem to me. I am curious as to the fencing configuration of the original poster.
Per above, I enabled dlm debug logging, and intriguingly enough get absolutely nothing. I suspect the thing is running away before getting to any task of substance.
I'll go ahead and paste in my cluster.conf in case it provides any insight, hostnames/IPs/login information sanitized.
<?xml version="1.0"?>
<cluster config_version="85" name="test-cluster">
<dlm log_debug="1"/>
<fence_daemon post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="node1-cluster" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="node1-bios"/>
</method>
</fence>
</clusternode>
<clusternode name="node2-cluster" nodeid="2" votes="1">
<fence>
<method name="1">
<device name="node2-bios"/>
</method>
</fence>
</clusternode>
<clusternode name="node3-cluster" nodeid="3" votes="1">
<fence>
<method name="1">
<device name="node3-bios"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman/>
<fencedevices>
<fencedevice agent="fence_drac" ipaddr="999.999.999.999" login="username" name="node1-bios" passwd="********"/>
<fencedevice agent="fence_drac" ipaddr="999.999.999.999" login="username" name="node2-bios" passwd="********"/>
<fencedevice agent="fence_drac" ipaddr="999.999.999.999" login="username" name="node3-bios" passwd="********"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="node1-only" ordered="0" restricted="1">
<failoverdomainnode name="node1-cluster" priority="1"/>
</failoverdomain>
<failoverdomain name="node2-only" restricted="1">
<failoverdomainnode name="node2-cluster" priority="1"/>
</failoverdomain>
<failoverdomain name="whole-cluster" ordered="0" restricted="0"/>
<failoverdomain name="test-nfs-servers" ordered="1" restricted="1">
<failoverdomainnode name="node1-cluster" priority="1"/>
<failoverdomainnode name="node2-cluster" priority="2"/>
</failoverdomain>
<failoverdomain name="node3-only" ordered="0" restricted="1">
<failoverdomainnode name="node3-cluster" priority="1"/>
</failoverdomain>
</failoverdomains>
<resources>
<clusterfs device="/dev/mapper/test-lvol0" force_unmount="1" fsid="7496" fstype="gfs" mountpoint="/test" name="testraid" options=""/>
<nfsexport name="test-nfs-export"/>
<nfsclient name="test-nfs-clients-128" options="async,rw" path="/test" target="999.999.999.999/255.255.255.254"/>
<ip address="999.999.999.999" monitor_link="1"/>
<nfsclient name="test-nfs-clients-246" options="async,rw" path="/test" target="999.999.999.999/255.255.255.240"/>
<nfsclient name="test-nfs-clients-227" options="async,rw" path="/test" target="999.999.999.999/255.255.255.192"/>
</resources>
<service autostart="1" domain="node1-only" name="node1-test-gfs">
<clusterfs ref="testraid"/>
</service>
<service autostart="1" domain="node2-only" name="node2-test-gfs">
<clusterfs ref="testraid"/>
</service>
<service autostart="1" domain="test-nfs-servers" name="test-nfs" recovery="relocate">
<clusterfs ref="testraid">
<nfsexport ref="test-nfs-export">
<nfsclient ref="test-nfs-clients-128"/>
<ip ref="999.999.999.999"/>
<nfsclient ref="test-nfs-clients-246"/>
<nfsclient ref="test-nfs-clients-227"/>
</nfsexport>
</clusterfs>
</service>
</rm>
</cluster>
I believe this has been solved, per the following: https://bugzilla.redhat.com/show_bug.cgi?id=472786#c4 Latest and greatest (cman-2.0.98-1.el5, openais-0.80.3-22.el5) appears to fix my problems. I think I hit this during 5.4 testing on ppc. I had run cpgx on the cluster and after stopping it and a few minutes passed one node started spewing oom-killer messages: [root@doral-p1 cpgx]# dlm_send invoked oom-killer: gfp_mask=0xd0, order=0, oomki lladj=0 Call Trace: [C00000007FCDB3A0] [C000000000010434] .show_stack+0x68/0x1b0 (unreliable) [C00000007FCDB440] [C0000000000C2918] .out_of_memory+0xa8/0x3c0 [C00000007FCDB500] [C0000000000C5E84] .__alloc_pages+0x270/0x33c [C00000007FCDB5F0] [C0000000000EEC28] .cache_grow+0x198/0x4e0 [C00000007FCDB6C0] [C0000000000EF3A8] .cache_alloc_refill+0x1e4/0x264 [C00000007FCDB770] [C0000000000EFA50] .kmem_cache_alloc+0xac/0xd8 [C00000007FCDB800] [C00000000032E7A4] .sock_alloc_inode+0x28/0x84 [C00000007FCDB880] [C00000000011E1C4] .alloc_inode+0x50/0x1ec [C00000007FCDB910] [C00000000011F78C] .new_inode+0x30/0xd8 [C00000007FCDB9A0] [C00000000033091C] .sock_alloc+0x28/0x94 [C00000007FCDBA20] [C000000000330AD0] .__sock_create+0x148/0x3a0 [C00000007FCDBAE0] [D000000000CEEB44] .tcp_connect_to_sock+0x98/0x2a0 [dlm] [C00000007FCDBCA0] [D000000000CEF1D0] .process_send_sockets+0x74/0x2cc [dlm] [C00000007FCDBD50] [C000000000082448] .run_workqueue+0xdc/0x168 [C00000007FCDBDF0] [C0000000000831B4] .worker_thread+0x12c/0x19c [C00000007FCDBEE0] [C000000000087CB0] .kthread+0x128/0x178 [C00000007FCDBF90] [C0000000000275CC] .kernel_thread+0x4c/0x68 Mem-info: Node 0 DMA per-cpu: cpu 0 hot: high 6, batch 1 used:0 cpu 0 cold: high 2, batch 1 used:1 cpu 1 hot: high 6, batch 1 used:3 cpu 1 cold: high 2, batch 1 used:1 Node 0 DMA32 per-cpu: empty Node 0 Normal per-cpu: empty Node 0 HighMem per-cpu: empty Free pages: 5760kB (0kB HighMem) Active:436 inactive:0 dirty:0 writeback:0 unstable:0 free:90 slab:31641 mapped-f ile:58 mapped-anon:362 pagetables:152 Node 0 DMA free:5760kB min:5760kB low:7168kB high:8640kB active:25856kB inactive :0kB present:2097152kB pages_scanned:148902 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 Node 0 DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0 kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present: 0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 HighMem free:0kB min:2048kB low:2048kB high:2048kB active:0kB inactive:0k B present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB 0*8192kB 0 *16384kB = 5760kB Node 0 DMA32: empty Node 0 Normal: empty Node 0 HighMem: empty 436 pagecache pages Swap cache: add 2469, delete 2107, find 19/48, race 0+3 Free swap = 3977984kB Total swap = 4128640kB Free swap: 3977984kB 32768 pages of RAM 350 reserved pages 899 pages shared 362 pages swap cached Out of memory: Killed process 2337 (ricci). Running on kernel 2.6.18-150.el5. *** This bug has been marked as a duplicate of bug 508829 *** |