Hide Forgot
Description of problem: When more than 2 nodes are booted and join the rgmanager service group (using a CMAN cluster), some of the nodes seem to not detect and/or receive the OOB service-join message on the cluster socket. The node joining the service group as well as all other nodes which receive the message go into 'D' state -- waiting for the nodes which "seem" to have missed the join-request. * This doesn't happen with other magma applications while using the sm plugin For instance, circleping works fine - regardless of the node count, or so it seems. * This doesn't happen when using the cman plugin (instead of the sm plugin). * This doesn't happen when using the gulm plugin. * This doesn't happen all the time. Running with debug logging enabled seems to make rgmanager join the service group correctly, which doesn't make sense. If it was a timing issue (preventing the receipt of the OOB messages), slowing rgmanager down should make the problem worse, not better. * This doesn't happen with multithreaded applications which access the service manager directly (e.g. clvmd). Still gathering more information.
Having a dedicated thread waiting for *only* cluster events seems to fix this. Will do slightly more testing and commit changes.
Turns out it only "appeared" on the SM plugin for some unknown reason. In reality, it was caused by the control thread entering accept(2) and being blocked because the listen file descriptors were not correctly set O_NONBLOCK.