Bug 1015466

Summary: [RHSC] Volume creation from the Console fails, with a popup which says "network error while communicating with host"
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Shruti Sampat <ssampat>
Component: rhscAssignee: Aravinda VK <avishwan>
Status: CLOSED CURRENTRELEASE QA Contact: Shruti Sampat <ssampat>
Severity: high Docs Contact:
Priority: high    
Version: 2.1CC: dpati, dtsang, knarra, mmahoney, pprakash, rhs-bugs, sabose, sdharane
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 2.1.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-25 11:06:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
screenshot
none
screenshot 2
none
engine logs
none
/var/log/vdsm/vdsm.log
none
glusterd logs none

Description Shruti Sampat 2013-10-04 10:34:21 UTC
Created attachment 807563 [details]
screenshot

Description of problem:
------------------------
Creation of volume from the Console, fails with an error pop-up which says the following - 

Error while executing action Create Gluster Volume: Network error during communication with the Host.

All nodes were up and running while this happened. When I tried again, the volume creation again failed with the error that the brick specified for this volume, is already part of another volume.

The following is seen in the vdsm logs -

Thread-2561::ERROR::2013-10-04 05:45:31,443::BindingXMLRPC::990::vds::(wrapper) vdsm exception occured
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 979, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 53, in wrapper
    rv = func(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 84, in volumeCreate
    transportList)
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in glusterVolumeCreate
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
GlusterVolumeCreateFailedException: <unprintable GlusterVolumeCreateFailedException object>


The following is seen in the engine logs -

2013-10-04 05:45:32,114 ERROR [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] (ajp-/127.0.0.1:8702-3) Command CreateGlusterVolumeVDS execution failed. Exception: VDSNetworkException: 
org.apache.xmlrpc.XmlRpcException: <type 'exceptions.TypeError'>:sequence item 0: expected string, NoneType found
2013-10-04 05:45:32,114 INFO  [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] (ajp-/127.0.0.1:8702-3) FINISH, CreateGlusterVolumeVDSCommand, log id: 3caa49d9
2013-10-04 05:45:32,114 ERROR [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] (ajp-/127.0.0.1:8702-3) Command org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand throw Vdc Bll excepti
on. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.TypeError'>:sequence item 0: expected string, NoneT
ype found (Failed with error VDS_NETWORK_ERROR and code 5022)


Version-Release number of selected component (if applicable):
Red Hat Storage Console Version: 2.1.1-0.0.2.master.el6ev 

How reproducible:
Intermittent

Steps to Reproduce:
1. Try to create a pure distribute volume from the Console.
2. After adding the bricks click on OK in the Create Volume dialog.

Actual results:
Volume creation fails with the above explained error.

Expected results:
Volume creation should succeed.

Additional info:
Find attached screenshots of both errors and engine logs, vdsm logs and glusterd logs.

Comment 1 Shruti Sampat 2013-10-04 10:35:30 UTC
Created attachment 807565 [details]
screenshot 2

Comment 2 Shruti Sampat 2013-10-04 10:40:13 UTC
Created attachment 807573 [details]
engine logs

Comment 3 Shruti Sampat 2013-10-04 10:43:19 UTC
Created attachment 807582 [details]
/var/log/vdsm/vdsm.log

Comment 4 Shruti Sampat 2013-10-04 10:44:07 UTC
Created attachment 807584 [details]
glusterd logs

Comment 6 Aravinda VK 2013-10-10 11:00:31 UTC
Network error if no up servers found. Sahina please confirm.

Second issue is expected behavior from glusterfs that if you used a directory as brick then gluster will not allow to create volume with the same brick.(vdsm logs shows the same)

Comment 7 Sahina Bose 2013-10-23 06:33:19 UTC
No, the VDS_NETWORK_ERROR is not related to UP server.

This is in the log:
2013-10-04 04:51:21,725 ERROR [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] (ajp-/127.0.0.1:8702-10) Command CreateGlusterVolumeVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.TypeError'>:sequence item 0: expected string, NoneType found


related to : GlusterVolumeCreateFailedException: <unprintable GlusterVolumeCreateFailedException object>

Comment 8 Dusmant 2013-11-25 09:44:29 UTC
This issue is not seen in CB8. If it's not seen in CB9, we can close it.

Comment 9 Shruti Sampat 2013-11-25 11:06:04 UTC
Not able to recreate in CB9. Closing the BZ.