Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1044110 - [Scale] 64 Brick Volume Create Failed Message - Volume Still Created
Summary: [Scale] 64 Brick Volume Create Failed Message - Volume Still Created
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: rhsc
Version: 2.1
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: ---
Assignee: Timothy Asir
QA Contact: storage-qa-internal@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-17 20:09 UTC by Matt Mahoney
Modified: 2015-09-01 13:53 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-09-01 13:53:18 UTC
Target Upstream Version:

Attachments (Terms of Use)
Create Volume Fail Message (6.18 MB, application/x-gzip)
2013-12-17 20:11 UTC, Matt Mahoney 		no flags Details
Add an attachment (proposed patch, testcase, etc.)

Description Matt Mahoney 2013-12-17 20:09:04 UTC 
Description of problem:
While creating a 64 brick volume that consisted of 8-servers with 8-bricks on each server, the following error message happened:


2013-Dec-17, 14:13 Gluster Volume Vol-6 created.
2013-Dec-17, 14:12 Creation of Gluster Volume Vol-6 failed.

Note the failed message (also appears to be such in the engine.log), but the volume was created. Validated via glusterfs that the volume was created, and that the volume was started/stopped with no issues


Version-Release number of selected component (if applicable):
cb12

How reproducible:
Happened 1 time out of 8 created volumes

Steps to Reproduce:
1. Create 64 brick volume (8-servers, 8-bricks on each server)
2.
3.

Actual results:
Creation of Gluster Volume <volumeName> failed.

Expected results:
No error message

Additional info:
Comment 1 Matt Mahoney 2013-12-17 20:11:47 UTC 
Created attachment 837938 [details]
Create Volume Fail Message
Comment 3 Kanagaraj 2013-12-18 05:33:37 UTC 
VDSM Error 

Thread-30560::DEBUG::2013-12-17 13:29:14,855::BindingXMLRPC::984::vds::(wrapper) client [10.16.159.68]::call volumeCreate with ('Vol-1', ['10.16.159.41:/bricks/b1', '10.16.159.41:/bricks/b2', '10.16.159.41:/bricks/b3', '10.16.159.41:/bricks/b4', '10.16.159.41:/bricks/b5', '10.16.159.41:/bricks/b6', '10.16.159.41:/bricks/b7', '10.16.159.41:/bricks/b8', '10.16.159.23:/bricks/b1', '10.16.159.23:/bricks/b2', '10.16.159.23:/bricks/b3', '10.16.159.23:/bricks/b4', '10.16.159.23:/bricks/b5', '10.16.159.23:/bricks/b6', '10.16.159.23:/bricks/b7', '10.16.159.23:/bricks/b8', '10.16.159.7:/bricks/b1', '10.16.159.7:/bricks/b2', '10.16.159.7:/bricks/b3', '10.16.159.7:/bricks/b4', '10.16.159.7:/bricks/b5', '10.16.159.7:/bricks/b6', '10.16.159.7:/bricks/b7', '10.16.159.7:/bricks/b8', '10.16.159.106:/bricks/b1', '10.16.159.106:/bricks/b2', '10.16.159.106:/bricks/b3', '10.16.159.106:/bricks/b4', '10.16.159.106:/bricks/b5', '10.16.159.106:/bricks/b6', '10.16.159.106:/bricks/b7', '10.16.159.106:/bricks/b8', '10.16.159.67:/bricks/b1', '10.16.159.67:/bricks/b2', '10.16.159.67:/bricks/b3', '10.16.159.67:/bricks/b4', '10.16.159.67:/bricks/b5', '10.16.159.67:/bricks/b6', '10.16.159.67:/bricks/b7', '10.16.159.67:/bricks/b8', '10.16.159.200:/bricks/b1', '10.16.159.200:/bricks/b2', '10.16.159.200:/bricks/b3', '10.16.159.200:/bricks/b4', '10.16.159.200:/bricks/b5', '10.16.159.200:/bricks/b6', '10.16.159.200:/bricks/b7', '10.16.159.200:/bricks/b8', '10.16.159.118:/bricks/b1', '10.16.159.118:/bricks/b2', '10.16.159.118:/bricks/b3', '10.16.159.118:/bricks/b4', '10.16.159.118:/bricks/b5', '10.16.159.118:/bricks/b6', '10.16.159.118:/bricks/b7', '10.16.159.118:/bricks/b8', '10.16.159.43:/bricks/b1', '10.16.159.43:/bricks/b2', '10.16.159.43:/bricks/b3', '10.16.159.43:/bricks/b4', '10.16.159.43:/bricks/b5', '10.16.159.43:/bricks/b6', '10.16.159.43:/bricks/b7', '10.16.159.43:/bricks/b8'], 0, 0, ['TCP'], True) {} flowID [42792772]
Thread-30560::ERROR::2013-12-17 13:29:15,007::BindingXMLRPC::1000::vds::(wrapper) vdsm exception occured
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 53, in wrapper
    rv = func(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 85, in volumeCreate
    transportList, force)
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in glusterVolumeCreate
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
GlusterVolumeCreateFailedException: Volume create failed
error: Staging failed on 10.16.159.41. Error: Failed to create brick directory for brick 10.16.159.41:/bricks/b1. Reason : No such file or directory 
Staging failed on 10.16.159.67. Error: Failed to create brick directory for brick 10.16.159.67:/bricks/b1. Reason : No such file or directory 
Staging failed on 10.16.159.23. Error: Failed to create brick directory for brick 10.16.159.23:/bricks/b1. Reason : No such file or directory 
Staging failed on 10.16.159.106. Error: Failed to create brick directory for brick 10.16.159.106:/bricks/b1. Reason : No such file or directory 
Staging failed on 10.16.159.7. Error: Failed to create brick directory for brick 10.16.159.7:/bricks/b1. Reason : No such file or directory 
Staging failed on 10.16.159.118. Error: Failed to create brick directory for brick 10.16.159.118:/bricks/b1. Reason : No such file or directory 
Staging failed on 10.16.159.200. Error: Failed to create brick directory for brick 10.16.159.200:/bricks/b1. Reason : No such file or directory 
Staging failed on 10.16.159.43. Error: Failed to create brick directory for brick 10.16.159.43:/bricks/b1. Reason : No such file or directory 
return code: 115


Gluster Error

[2013-12-17 18:29:15.001523] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.41. Error: Failed to create brick directory for brick 10.16.159.41:/bricks/b1. Reason : No such file or directory 
[2013-12-17 18:29:15.001691] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.67. Error: Failed to create brick directory for brick 10.16.159.67:/bricks/b1. Reason : No such file or directory 
[2013-12-17 18:29:15.001902] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.23. Error: Failed to create brick directory for brick 10.16.159.23:/bricks/b1. Reason : No such file or directory 
[2013-12-17 18:29:15.002295] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.106. Error: Failed to create brick directory for brick 10.16.159.106:/bricks/b1. Reason : No such file or directory 
[2013-12-17 18:29:15.002350] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.7. Error: Failed to create brick directory for brick 10.16.159.7:/bricks/b1. Reason : No such file or directory 
[2013-12-17 18:29:15.002383] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.118. Error: Failed to create brick directory for brick 10.16.159.118:/bricks/b1. Reason : No such file or directory 
[2013-12-17 18:29:15.002416] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.200. Error: Failed to create brick directory for brick 10.16.159.200:/bricks/b1. Reason : No such file or directory 
[2013-12-17 18:29:15.002715] E [glusterd-syncop.c:102:gd_collate_errors] 0-: Staging failed on 10.16.159.43. Error: Failed to create brick directory for brick 10.16.159.43:/bricks/b1. Reason : No such file or directory 
[2013-12-17 18:31:36.763529] I [glusterd-handler.c:1018:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2013-12-17 18:31:37.089730] I [glusterd-handler.c:1073:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2013-12-17 18:36:29.805126] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=Vol-1 -o nfs.disable=off
[2013-12-17 18:36:53.196168] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=Vol-1 -o user.cifs=enable
[2013-12-17 18:37:13.281808] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=Vol-1 -o auth.allow=*
[2013-12-17 18:39:11.232694] I [rpc-clnt.c:977:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2013-12-17 18:39:11.232760] I [socket.c:3505:socket_init] 0-management: SSL support is NOT enabled
[2013-12-17 18:39:11.232769] I [socket.c:3520:socket_init] 0-management: using system polling thread
[2013-12-17 18:39:11.233336] I [socket.c:2235:socket_event_handler] 0-transport: disconnecting now
[2013-12-17 18:39:11.237452] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=Vol-1 --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
[2013-12-17 18:39:11.257430] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=Vol-1 --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
[2013-12-17 18:39:35.109709] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=Vol-1 --last=yes
[2013-12-17 18:39:35.124108] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=Vol-1 --last=yes
[2013-12-17 18:40:17.166131] E [glusterd-utils.c:3873:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/813402e4dc3236c4c4a378a286ac9746.socket error: Permission denied
[2013-12-17 18:40:17.166497] I [glusterd-utils.c:3907:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully
[2013-12-17 18:40:17.166610] I [glusterd-utils.c:3912:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully
[2013-12-17 18:40:17.166730] I [glusterd-utils.c:3917:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully
[2013-12-17 18:40:17.166883] I [glusterd-utils.c:3922:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully
[2013-12-17 18:40:17.166994] I [glusterd-utils.c:3927:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully
[2013-12-17 18:40:17.167117] I [glusterd-utils.c:3932:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully
[2013-12-17 18:40:17.167287] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=2236 max=0 total=0
[2013-12-17 18:40:17.167308] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=124 max=0 total=0
Comment 4 Dusmant 2015-09-01 13:53:18 UTC 
We are not planning to fix it. Hence closing this BZ. If you think, it's applicable for 3.x release and would have an impact on customer, pls. open up a new BZ with the appropriate version. Thanks, -Dusmant
Note You need to log in before you can comment on or make changes to this bug.