Description of problem: While creating a 64 brick volume that consisted of 8-servers with 8-bricks on each server, the following error message happened: 2013-Dec-17, 14:13 Gluster Volume Vol-6 created. 2013-Dec-17, 14:12 Creation of Gluster Volume Vol-6 failed. Note the failed message (also appears to be such in the engine.log), but the volume was created. Validated via glusterfs that the volume was created, and that the volume was started/stopped with no issues Version-Release number of selected component (if applicable): cb12 How reproducible: Happened 1 time out of 8 created volumes Steps to Reproduce: 1. Create 64 brick volume (8-servers, 8-bricks on each server) 2. 3. Actual results: Creation of Gluster Volume <volumeName> failed. Expected results: No error message Additional info:
Created attachment 837938 [details] Create Volume Fail Message
VDSM Error Thread-30560::DEBUG::2013-12-17 13:29:14,855::BindingXMLRPC::984::vds::(wrapper) client [10.16.159.68]::call volumeCreate with ('Vol-1', ['10.16.159.41:/bricks/b1', '10.16.159.41:/bricks/b2', '10.16.159.41:/bricks/b3', '10.16.159.41:/bricks/b4', '10.16.159.41:/bricks/b5', '10.16.159.41:/bricks/b6', '10.16.159.41:/bricks/b7', '10.16.159.41:/bricks/b8', '10.16.159.23:/bricks/b1', '10.16.159.23:/bricks/b2', '10.16.159.23:/bricks/b3', '10.16.159.23:/bricks/b4', '10.16.159.23:/bricks/b5', '10.16.159.23:/bricks/b6', '10.16.159.23:/bricks/b7', '10.16.159.23:/bricks/b8', '10.16.159.7:/bricks/b1', '10.16.159.7:/bricks/b2', '10.16.159.7:/bricks/b3', '10.16.159.7:/bricks/b4', '10.16.159.7:/bricks/b5', '10.16.159.7:/bricks/b6', '10.16.159.7:/bricks/b7', '10.16.159.7:/bricks/b8', '10.16.159.106:/bricks/b1', '10.16.159.106:/bricks/b2', '10.16.159.106:/bricks/b3', '10.16.159.106:/bricks/b4', '10.16.159.106:/bricks/b5', '10.16.159.106:/bricks/b6', '10.16.159.106:/bricks/b7', '10.16.159.106:/bricks/b8', '10.16.159.67:/bricks/b1', '10.16.159.67:/bricks/b2', '10.16.159.67:/bricks/b3', '10.16.159.67:/bricks/b4', '10.16.159.67:/bricks/b5', '10.16.159.67:/bricks/b6', '10.16.159.67:/bricks/b7', '10.16.159.67:/bricks/b8', '10.16.159.200:/bricks/b1', '10.16.159.200:/bricks/b2', '10.16.159.200:/bricks/b3', '10.16.159.200:/bricks/b4', '10.16.159.200:/bricks/b5', '10.16.159.200:/bricks/b6', '10.16.159.200:/bricks/b7', '10.16.159.200:/bricks/b8', '10.16.159.118:/bricks/b1', '10.16.159.118:/bricks/b2', '10.16.159.118:/bricks/b3', '10.16.159.118:/bricks/b4', '10.16.159.118:/bricks/b5', '10.16.159.118:/bricks/b6', '10.16.159.118:/bricks/b7', '10.16.159.118:/bricks/b8', '10.16.159.43:/bricks/b1', '10.16.159.43:/bricks/b2', '10.16.159.43:/bricks/b3', '10.16.159.43:/bricks/b4', '10.16.159.43:/bricks/b5', '10.16.159.43:/bricks/b6', '10.16.159.43:/bricks/b7', '10.16.159.43:/bricks/b8'], 0, 0, ['TCP'], True) {} flowID [42792772] Thread-30560::ERROR::2013-12-17 13:29:15,007::BindingXMLRPC::1000::vds::(wrapper) vdsm exception occured Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 53, in wrapper rv = func(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 85, in volumeCreate transportList, force) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in glusterVolumeCreate File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod raise convert_to_error(kind, result) GlusterVolumeCreateFailedException: Volume create failed error: Staging failed on 10.16.159.41. Error: Failed to create brick directory for brick 10.16.159.41:/bricks/b1. Reason : No such file or directory Staging failed on 10.16.159.67. Error: Failed to create brick directory for brick 10.16.159.67:/bricks/b1. Reason : No such file or directory Staging failed on 10.16.159.23. Error: Failed to create brick directory for brick 10.16.159.23:/bricks/b1. Reason : No such file or directory Staging failed on 10.16.159.106. Error: Failed to create brick directory for brick 10.16.159.106:/bricks/b1. Reason : No such file or directory Staging failed on 10.16.159.7. Error: Failed to create brick directory for brick 10.16.159.7:/bricks/b1. Reason : No such file or directory Staging failed on 10.16.159.118. Error: Failed to create brick directory for brick 10.16.159.118:/bricks/b1. Reason : No such file or directory Staging failed on 10.16.159.200. Error: Failed to create brick directory for brick 10.16.159.200:/bricks/b1. Reason : No such file or directory Staging failed on 10.16.159.43. Error: Failed to create brick directory for brick 10.16.159.43:/bricks/b1. Reason : No such file or directory return code: 115 Gluster Error [2013-12-17 18:29:15.001523] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.41. Error: Failed to create brick directory for brick 10.16.159.41:/bricks/b1. Reason : No such file or directory [2013-12-17 18:29:15.001691] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.67. Error: Failed to create brick directory for brick 10.16.159.67:/bricks/b1. Reason : No such file or directory [2013-12-17 18:29:15.001902] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.23. Error: Failed to create brick directory for brick 10.16.159.23:/bricks/b1. Reason : No such file or directory [2013-12-17 18:29:15.002295] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.106. Error: Failed to create brick directory for brick 10.16.159.106:/bricks/b1. Reason : No such file or directory [2013-12-17 18:29:15.002350] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.7. Error: Failed to create brick directory for brick 10.16.159.7:/bricks/b1. Reason : No such file or directory [2013-12-17 18:29:15.002383] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.118. Error: Failed to create brick directory for brick 10.16.159.118:/bricks/b1. Reason : No such file or directory [2013-12-17 18:29:15.002416] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.200. Error: Failed to create brick directory for brick 10.16.159.200:/bricks/b1. Reason : No such file or directory [2013-12-17 18:29:15.002715] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.43. Error: Failed to create brick directory for brick 10.16.159.43:/bricks/b1. Reason : No such file or directory [2013-12-17 18:31:36.763529] I [glusterd-handler.c:1018:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2013-12-17 18:31:37.089730] I [glusterd-handler.c:1073:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2013-12-17 18:36:29.805126] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=Vol-1 -o nfs.disable=off [2013-12-17 18:36:53.196168] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=Vol-1 -o user.cifs=enable [2013-12-17 18:37:13.281808] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=Vol-1 -o auth.allow=* [2013-12-17 18:39:11.232694] I [rpc-clnt.c:977:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2013-12-17 18:39:11.232760] I [socket.c:3505:socket_init] 0-management: SSL support is NOT enabled [2013-12-17 18:39:11.232769] I [socket.c:3520:socket_init] 0-management: using system polling thread [2013-12-17 18:39:11.233336] I [socket.c:2235:socket_event_handler] 0-transport: disconnecting now [2013-12-17 18:39:11.237452] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=Vol-1 --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd [2013-12-17 18:39:11.257430] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=Vol-1 --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd [2013-12-17 18:39:35.109709] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=Vol-1 --last=yes [2013-12-17 18:39:35.124108] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=Vol-1 --last=yes [2013-12-17 18:40:17.166131] E [glusterd-utils.c:3873:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/813402e4dc3236c4c4a378a286ac9746.socket error: Permission denied [2013-12-17 18:40:17.166497] I [glusterd-utils.c:3907:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully [2013-12-17 18:40:17.166610] I [glusterd-utils.c:3912:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully [2013-12-17 18:40:17.166730] I [glusterd-utils.c:3917:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully [2013-12-17 18:40:17.166883] I [glusterd-utils.c:3922:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully [2013-12-17 18:40:17.166994] I [glusterd-utils.c:3927:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully [2013-12-17 18:40:17.167117] I [glusterd-utils.c:3932:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully [2013-12-17 18:40:17.167287] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=2236 max=0 total=0 [2013-12-17 18:40:17.167308] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=124 max=0 total=0
We are not planning to fix it. Hence closing this BZ. If you think, it's applicable for 3.x release and would have an impact on customer, pls. open up a new BZ with the appropriate version. Thanks, -Dusmant