Description of problem: If on node failed I can't replace it, got key error. Version-Release number of selected component (if applicable): 0.7-15.fc16 What is the plan to replace an node which failure? Can't mount the volume if one node did not come up... Need quick solution.
See http://gluster.org/community/documentation//index.php/Gluster_3.2:_Brick_Restoration_-_Replace_Crashed_Server for guidance on replacing a failed node. If this doesn't work you can ask for help on IRC on #gluster @ freenode
To elaborate on Kaleb's answer... In HekaFS, each tenant effectively has their own volume assembled from the per-tenant directories on each brick. This is what allows different tenants to use different replication levels etc. (even though this feature isn't currently exposed through the management interfaces). What this means for replacing a failed node is that the GlusterFS instructions will work as far as getting the new node/brick(s) into the system, but each tenant must do their own self-heal. The GlusterFS instructions also fail to mention that a client will need to re-mount so it will know to connect to the new server instead of the old one. This is as true in HekaFS as it is in "vanilla" GlusterFS.
(In reply to comment #2) > To elaborate on Kaleb's answer... > > In HekaFS, each tenant effectively has their own volume assembled from the > per-tenant directories on each brick. This is what allows different tenants to > use different replication levels etc. (even though this feature isn't currently > exposed through the management interfaces). What this means for replacing a > failed node is that the GlusterFS instructions will work as far as getting the > new node/brick(s) into the system, but each tenant must do their own self-heal. > > The GlusterFS instructions also fail to mention that a client will need to > re-mount so it will know to connect to the new server instead of the old one. > This is as true in HekaFS as it is in "vanilla" GlusterFS. Hi, first thank you for the informations and support. Can you explain how I can start the self healing process for the tenants? Is this the same find command like in this how to: http://gluster.org/community/documentation/index.php/Gluster_3.2:_Triggering_Self-Heal_on_Replicate And what is more important, I add two more bricks to an volume and after remount we did not found andy folders which was on the first 2 bricks. Self healing is not working, I can mkdir the folder and the systems tells me that the folder exists but we have a lot of subfolders and can't mkdir to all this folders.
(In reply to comment #3) > Can you explain how I can start the self healing process for the tenants? Is > this the same find command like in this how to: > > http://gluster.org/community/documentation/index.php/Gluster_3.2:_Triggering_Self-Heal_on_Replicate That is correct, with the caveat that it must be done separately through each tenant mountpoint. > And what is more important, I add two more bricks to an volume and after > remount we did not found andy folders which was on the first 2 bricks. > Self healing is not working, I can mkdir the folder and the systems tells me > that the folder exists but we have a lot of subfolders and can't mkdir to all > this folders. If I recall our conversation on IRC correctly, you said that you had created a new volume with the same name as a previous one, but with more bricks. This seems a bit problematic with respect to things like brick UUIDs and xattr values. To diagnose, we'll need some more information, such as: * client and server logs (especially the embedded volfiles and messages from around each daemon's startup) * xattr values (anything with "gluster" in the name) from at least the per-tenant directories on each brick Then again, this seems to be a separate problem so perhaps it should be a separate bug. Mind if I clone this one?
Yes this would be fine, I have to recreate an Volume with same name, I did not have an option to expand my Volume,. client Volfile from hfs_mount: http://pastie.org/3480507 Volfile Node1: http://pastie.org/3480544 Volfile Node2: http://pastie.org/3480550 Volfile Node3: http://pastie.org/3480558 Volfile Node4: http://pastie.org/3480563
HekaFS will be merged into core GlusterFS