Hide Forgot
Description of problem: We have hit corruption in DHT layout. * DHT start range was found bigger than end range. * DHT hash range not well distributed among all the bricks This was observed after rebalance failure when 100% brick full occured. Version-Release number of selected component (if applicable): 3.7.1-11.el6rhs.x86_64 Additional info : 100% brick full occurred probably due to running rebalance with "force" command. Rebalance was executed after adding 4 new nodes with 5 brick each i.e a total of 20 new bricks. current volume size : 16 nodes, 5 bricks each i.e a total of 80 bricks Volume type : Distribute
Bipin, Can you update the brick sizes from all the nodes?
RCA: There was bug with weighted-rebalance option in version 3.7.1-11 where the sum of size of all the bricks were stored in an unsigned integer (uint32_t). For big clusters with larger size bricks like the current one where each brick size is 55TB (totaling to (55TB * 80 = 4.8PB) ), the value will overflow causing incorrect chunk computation, giving rise to overflowing layout every few bricks We had hit a similar bug before here: https://bugzilla.redhat.com/show_bug.cgi?id=1281946 This was fixed in 3.1.2 Patch: https://code.engineering.redhat.com/gerrit/#/c/64630/ For workaround customer can turn the weighted-rebalance off and remount all the clients or upgrade to 3.1.2.
For the new dir creation we saw that dht layout was getting proper hash range sometimes with gs9 bricks with no hash range as it is almost 100%, but we did see hash layout corruption for few of them. Not sure what is the condition when dht stops giving hash range to new dirs, Might be min-free disk, but not sure. We tried workaround for now to off weighted-rebalance. As of now weighted-rebalance off is working fine. We did lookup from new mounts(all the old mounts were unmounted) and saw layout getting rectified. We started lookup recursively on all the directory in order to fix the layout for all. I will close this bug as the fix is already available in newer releases.
This should be marked as closed, current release.