Red Hat Bugzilla – Bug 173633
cman/sm nodeid lookup fails
Last modified: 2009-04-16 16:30:42 EDT
Description of problem: System running 2.6.9-22 kernel sitting idle went into kdb with Assertion failed on line 52 of file cluster/cman/sm_misc.c Attached typescript file has kdb: bt, sr -t, ps, dmesg output. Version-Release number of selected component (if applicable): kernel = 2.6.9-22 cman-1.0.2-0 cman-devel-1.0.2-0 How reproducible: Not sure. We've only seen it once to date and the system was not doing anything. Testers came in and found it kdb. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Created attachment 121241 [details] ascii file output from kdb, see above description
Could you verify that cluster.conf was the same on all nodes?, and a copy of that may be helpful to see, along with 'cman_tool nodes' from one of the other nodes if they are still running. This assertion failure may indicate some sort of internal consistency problem within cman: the sm portion is looking up a nodeid that the cnxman portion doesn't know about, which shouldn't be possible. If the assignment of nodeid's to nodes was changing while the cluster was running (different versions of cluster.conf on the nodes possibly), that might lead to this kind of error. If untypically large nodeids are being used, that may point toward the cnxman code that dynamically increases the standard node arrays. This assertion failure was reported once before to me in an email (in May by Dan Phung) and he had been updating cluster.conf on some nodes. I'm adding a printk to the code to provide a little more information in the assertion message if this happens again.
waiting for someone to see this again and report with the additional info from the panic