Hide Forgot
Description of problem: Rebalance starts on a volume which has multiple nodes participating with bricks and glusterd is down on one of these nodes. However if the status if executed for rebalance on the same volume, it shows the status as FAILED. Version-Release number of selected component (if applicable): How reproducible: Ocassionally. Mostly it gives the error saying "Rebalance cannot be stared, One of the bricks is down". Steps to Reproduce: 1. Create a two node cluster 2. Create a volume say vol1 with bricks on both the nodes 3. Bring the glusterd down on one of the nodes using "service glusterd stop" 4. Start rebalance on the volume vol1 Actual results: Sometimes the start rebalance is successful and the status of rebalance shows FAILED. Expected results: Start rebalance should always fail in first stage if one of the participating nodes has glusterd down. Additional info:
Per bug triage discussion with Shanks and Dusmant, removing it from corbett
The issue here is with the order of daemonizing and graph initialization in the glusterfsd process. The fetching of the volfiles and the graph initialization is done after the process is daemonized. When glusterd starts the rebalance process, it returns after the process has daemonized assuming (correctly) that the process started and returns that starting the rebalance process was successful. But, in this particular case the graph initialization of the rebalance process fails as it cannot connect to the brick on the downed peer (this failure occurs even if just glusterd is down, as the brick port cannot be obtained by the client xlator). Since, rebalance requires all DHT subvolumes to be online, the process kills itself. This leads to the rebalance status showing as failed almost immediately. This is similar to rebalance ending up in failed status when a peer goes down during rebalance. This could be fixed in two ways, 1. Make sure that rebalance process has correctly initialized its graph and is connected to all the bricks before returning success. This is quite hard to, and probably requires new tolling to be done to support this approach, 2. Check if all the peers involved and the bricks involved are online during the staging of rebalance start. This is comparatively easier to do, as we already have a mechanism to check volume quorum thanks to volume snapshots. But this is also big change involving significant code changes as the volume quorum framework as it is present is kindof tied up with the snapshot. Removing devel_ack and the Denali label as solving this issue for the Denali is not easy or straight forward.
Cloning this to 3.1. to be fixed in future release.