Attachment 856014 Details for Bug 1058253 – VDSM log Correct

VDSM log Correct

vdsm.out.txt (text/plain), 132.80 KB, created by Jon Archer on 2014-01-27 11:08:32 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Jon Archer

Created: 2014-01-27 11:08:32 UTC

Size: 132.80 KB

patch

obsolete

>
>libvirtEventLoop::DEBUG::2014-01-27 10:25:21,518::vm::2373::vm.Vm::(onReboot) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::reboot event
>libvirtEventLoop::DEBUG::2014-01-27 10:25:21,535::vm::5098::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::event Shutdown detail 0 opaque None
>VM Channels Listener::ERROR::2014-01-27 10:25:21,606::vmChannels::53::vds::(_handle_event) Received 00000019 on fileno 65
>VM Channels Listener::DEBUG::2014-01-27 10:25:21,610::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 65.
>VM Channels Listener::DEBUG::2014-01-27 10:25:21,611::guestIF::147::vm.Vm::(_connect) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::Attempting connection to /var/lib/libvirt/qemu/channels/539cc03c-a3a8-4e56-9330-2f729667dde7.com.redhat.rhevm.vdsm
>VM Channels Listener::DEBUG::2014-01-27 10:25:21,611::guestIF::158::vm.Vm::(_connect) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::Failed to connect to /var/lib/libvirt/qemu/channels/539cc03c-a3a8-4e56-9330-2f729667dde7.com.redhat.rhevm.vdsm with 111
>libvirtEventLoop::DEBUG::2014-01-27 10:25:21,777::vm::5098::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::event Stopped detail 0 opaque None
>libvirtEventLoop::INFO::2014-01-27 10:25:21,777::vm::2308::vm.Vm::(_onQemuDeath) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::underlying process disconnected
>libvirtEventLoop::INFO::2014-01-27 10:25:21,777::vm::4561::vm.Vm::(releaseVm) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::Release VM resources
>libvirtEventLoop::DEBUG::2014-01-27 10:25:21,782::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found: no domain with matching uuid '539cc03c-a3a8-4e56-9330-2f729667dde7'
>libvirtEventLoop::DEBUG::2014-01-27 10:25:21,783::sampling::292::vm.Vm::(stop) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::Stop statistics collection
>libvirtEventLoop::DEBUG::2014-01-27 10:25:21,783::vmChannels::205::vds::(unregister) Delete fileno 65 from listener.
>Thread-1806::DEBUG::2014-01-27 10:25:21,783::sampling::323::vm.Vm::(run) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::Stats thread finished
>libvirtEventLoop::DEBUG::2014-01-27 10:25:21,784::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found: no domain with matching uuid '539cc03c-a3a8-4e56-9330-2f729667dde7'
>libvirtEventLoop::WARNING::2014-01-27 10:25:21,784::clientIF::332::vds::(teardownVolumePath) Drive is not a vdsm image: VOLWM_CHUNK_MB:1024 VOLWM_CHUNK_REPLICATE_MULT:2 VOLWM_FREE_PCT:50 _blockDev:False _checkIoTuneCategories:<bound method Drive._checkIoTuneCategories of <vm.Drive object at 0x7fd6ec13d690>> _customize:<bound method Drive._customize of <vm.Drive object at 0x7fd6ec13d690>> _deviceXML:<disk device="cdrom" snapshot="no" type="file"><address bus="1" controller="0" target="0" type="drive" unit="0"/><source file="" startupPolicy="optional"/><target bus="ide" dev="hdc"/><readonly/><serial></serial></disk> _makeName:<bound method Drive._makeName of <vm.Drive object at 0x7fd6ec13d690>> _setExtSharedState:<bound method Drive._setExtSharedState of <vm.Drive object at 0x7fd6ec13d690>> _validateIoTuneParams:<bound method Drive._validateIoTuneParams of <vm.Drive object at 0x7fd6ec13d690>> address:{u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'} alias:ide0-1-0 apparentsize:0 blockDev:False cache:none conf:{'status': 'Up', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.5.0', 'vmId': '539cc03c-a3a8-4e56-9330-2f729667dde7', 'pid': '8461', 'memGuaranteedSize': 1024, 'timeOffset': -2L, 'keyboardLayout': 'en-us', 'displayPort': u'5900', 'displaySecurePort': u'5901', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Conroe', 'smp': '1', 'pauseCode': 'NOERR', 'clientIp': '', 'nicModel': 'rtl8139,pv', 'smartcardEnable': 'false', 'kvmEnable': 'true', 'pitReinjection': 'false', 'transparentHugePages': 'true', 'displayNetwork': 'ovirtmgmt', 'devices': [{'target': 1048576, 'alias': u'balloon0', 'specParams': {'model': 'virtio'}, 'deviceId': '3d3208b0-ac7b-4f19-8fc1-963995c38eec', 'address': {u'slot': u'0x07', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'device': 'memballoon', 'type': 'balloon'}, {'index': '0', 'alias': u'scsi0', 'specParams': {}, 'deviceId': '06187ab5-9bc5-4029-9c61-7a23b197eb86', 'address': {u'slot': u'0x04', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'alias': u'video0', 'specParams': {'vram': '32768', 'ram': '65536', 'heads': '1'}, 'deviceId': '7b4a2393-d5a2-4c0a-bc1f-ceae212ae4a1', 'address': {u'slot': u'0x02', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'device': 'qxl', 'type': 'video'}, {'nicModel': 'pv', 'macAddr': '00:1a:4a:2d:98:ee', 'linkActive': True, 'network': 'ovirtmgmt', 'alias': u'net0', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': '7f2863e8-ebc1-4c36-a1fe-9e427260e50e', 'address': {u'slot': u'0x03', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'device': 'bridge', 'type': 'interface', 'name': u'vnet0'}, {'index': '2', 'iface': 'ide', 'name': u'hdc', 'alias': u'ide0-1-0', 'address': {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, 'specParams': {'path': ''}, 'readonly': 'True', 'deviceId': '265fd90e-7705-49a0-be91-275d60665560', 'path': '', 'device': 'cdrom', 'shared': 'false', 'type': 'disk'}, {'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'volumeInfo': {'domainID': 'c074a6e3-87be-445e-890f-601c215ba320', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/images/748aef1d-5d81-4d67-ad4f-ee2e5b258606/dd3a97b2-1c41-4e7f-b9ca-538061b2f827', 'volumeID': 'dd3a97b2-1c41-4e7f-b9ca-538061b2f827', 'leasePath': '/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/images/748aef1d-5d81-4d67-ad4f-ee2e5b258606/dd3a97b2-1c41-4e7f-b9ca-538061b2f827.lease', 'imageID': '748aef1d-5d81-4d67-ad4f-ee2e5b258606'}, 'index': 0, 'iface': 'virtio', 'apparentsize': '107374182400', 'alias': u'virtio-disk0', 'imageID': '748aef1d-5d81-4d67-ad4f-ee2e5b258606', 'readonly': 'False', 'shared': 'false', 'truesize': '1494982656', 'type': 'disk', 'domainID': 'c074a6e3-87be-445e-890f-601c215ba320', 'reqsize': '0', 'format': 'raw', 'deviceId': '748aef1d-5d81-4d67-ad4f-ee2e5b258606', 'poolID': 'a13a367c-accd-451a-a2f3-4c83660f1719', 'device': 'disk', 'path': '/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/images/748aef1d-5d81-4d67-ad4f-ee2e5b258606/dd3a97b2-1c41-4e7f-b9ca-538061b2f827', 'propagateErrors': 'off', 'optional': 'false', 'name': u'vda', 'bootOrder': u'1', 'volumeID': 'dd3a97b2-1c41-4e7f-b9ca-538061b2f827', 'specParams': {}, 'volumeChain': [{'domainID': 'c074a6e3-87be-445e-890f-601c215ba320', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/images/748aef1d-5d81-4d67-ad4f-ee2e5b258606/dd3a97b2-1c41-4e7f-b9ca-538061b2f827', 'volumeID': 'dd3a97b2-1c41-4e7f-b9ca-538061b2f827', 'leasePath': '/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/images/748aef1d-5d81-4d67-ad4f-ee2e5b258606/dd3a97b2-1c41-4e7f-b9ca-538061b2f827.lease', 'imageID': '748aef1d-5d81-4d67-ad4f-ee2e5b258606'}]}, {'device': u'usb', 'alias': u'usb0', 'type': 'controller', 'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x2'}}, {'device': u'ide', 'alias': u'ide0', 'type': 'controller', 'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x1'}}, {'device': u'virtio-serial', 'alias': u'virtio-serial0', 'type': 'controller', 'address': {u'slot': u'0x05', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}}, {'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}, {'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}, {'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}], 'custom': {'device_a102b644-e920-4e9a-a18e-c3c5f9c71e16device_982a0aaa-95b5-4230-b07f-a91755b27592device_9dc9f969-b730-4bc2-b66c-001c93d9e6addevice_064b3e6d-4c45-4695-be2a-32b93cda290cdevice_e78cd938-60d5-4918-97c5-9b5f244bd4e2': 'VmDevice {vmId=539cc03c-a3a8-4e56-9330-2f729667dde7, deviceId=e78cd938-60d5-4918-97c5-9b5f244bd4e2, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}', 'device_a102b644-e920-4e9a-a18e-c3c5f9c71e16device_982a0aaa-95b5-4230-b07f-a91755b27592': 'VmDevice {vmId=539cc03c-a3a8-4e56-9330-2f729667dde7, deviceId=982a0aaa-95b5-4230-b07f-a91755b27592, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}', 'device_a102b644-e920-4e9a-a18e-c3c5f9c71e16': 'VmDevice {vmId=539cc03c-a3a8-4e56-9330-2f729667dde7, deviceId=a102b644-e920-4e9a-a18e-c3c5f9c71e16, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}', 'device_a102b644-e920-4e9a-a18e-c3c5f9c71e16device_982a0aaa-95b5-4230-b07f-a91755b27592device_9dc9f969-b730-4bc2-b66c-001c93d9e6ad': 'VmDevice {vmId=539cc03c-a3a8-4e56-9330-2f729667dde7, deviceId=9dc9f969-b730-4bc2-b66c-001c93d9e6ad, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}', 'device_a102b644-e920-4e9a-a18e-c3c5f9c71e16device_982a0aaa-95b5-4230-b07f-a91755b27592device_9dc9f969-b730-4bc2-b66c-001c93d9e6addevice_064b3e6d-4c45-4695-be2a-32b93cda290c': 'VmDevice {vmId=539cc03c-a3a8-4e56-9330-2f729667dde7, deviceId=064b3e6d-4c45-4695-be2a-32b93cda290c, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}'}, 'vmType': 'kvm', 'memSize': 1024, 'displayIp': '192.168.1.107', 'spiceSecureChannels': 'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard', 'smpCoresPerSocket': '1', 'vmName': 'wcsmail01', 'display': 'qxl', 'nice': '0'} createXmlElem:<bound method Drive.createXmlElem of <vm.Drive object at 0x7fd6ec13d690>> device:cdrom deviceId:265fd90e-7705-49a0-be91-275d60665560 drv:raw extSharedState:none getLeasesXML:<bound method Drive.getLeasesXML of <vm.Drive object at 0x7fd6ec13d690>> getNextVolumeSize:<bound method Drive.getNextVolumeSize of <vm.Drive object at 0x7fd6ec13d690>> getXML:<bound method Drive.getXML of <vm.Drive object at 0x7fd6ec13d690>> hasVolumeLeases:False iface:ide index:2 isDiskReplicationInProgress:<bound method Drive.isDiskReplicationInProgress of <vm.Drive object at 0x7fd6ec13d690>> log:<logUtils.SimpleLogAdapter instance at 0x7fd6ec151170> name:hdc networkDev:False path: readonly:True reqsize:0 serial: shared:false specParams:{'path': ''} transientDisk:False truesize:0 type:cdrom volExtensionChunk:1024 watermarkLimit:536870912
>Traceback (most recent call last):
>  File "/usr/share/vdsm/clientIF.py", line 326, in teardownVolumePath
>    res = self.irs.teardownImage(drive['domainID'],
>  File "/usr/share/vdsm/vm.py", line 1493, in __getitem__
>    raise KeyError(key)
>KeyError: 'domainID'
>libvirtEventLoop::INFO::2014-01-27 10:25:21,788::logUtils::44::dispatcher::(wrapper) Run and protect: teardownImage(sdUUID='c074a6e3-87be-445e-890f-601c215ba320', spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', imgUUID='748aef1d-5d81-4d67-ad4f-ee2e5b258606', volUUID=None)
>libvirtEventLoop::INFO::2014-01-27 10:25:21,789::logUtils::47::dispatcher::(wrapper) Run and protect: teardownImage, Return response: None
>libvirtEventLoop::INFO::2014-01-27 10:25:21,790::logUtils::44::dispatcher::(wrapper) Run and protect: inappropriateDevices(thiefId='539cc03c-a3a8-4e56-9330-2f729667dde7')
>libvirtEventLoop::INFO::2014-01-27 10:25:21,795::logUtils::47::dispatcher::(wrapper) Run and protect: inappropriateDevices, Return response: None
>libvirtEventLoop::DEBUG::2014-01-27 10:25:21,801::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found: no domain with matching uuid '539cc03c-a3a8-4e56-9330-2f729667dde7'
>libvirtEventLoop::DEBUG::2014-01-27 10:25:21,801::vm::2725::vm.Vm::(setDownStatus) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::Changed state to Down: User shut down
>libvirtEventLoop::DEBUG::2014-01-27 10:25:21,801::sampling::292::vm.Vm::(stop) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::Stop statistics collection
>libvirtEventLoop::DEBUG::2014-01-27 10:25:21,806::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found: no domain with matching uuid '539cc03c-a3a8-4e56-9330-2f729667dde7'
>Thread-2162::DEBUG::2014-01-27 10:25:22,348::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call vmGetStats with ('539cc03c-a3a8-4e56-9330-2f729667dde7',) {}
>Thread-2162::DEBUG::2014-01-27 10:25:22,349::BindingXMLRPC::977::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Down', 'hash': '5105513022204860002', 'exitMessage': 'User shut down', 'vmId': '539cc03c-a3a8-4e56-9330-2f729667dde7', 'timeOffset': -2L, 'exitCode': 0}]}
>Thread-2163::DEBUG::2014-01-27 10:25:22,361::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call vmDestroy with ('539cc03c-a3a8-4e56-9330-2f729667dde7',) {}
>Thread-2163::INFO::2014-01-27 10:25:22,361::API::327::vds::(destroy) vmContainerLock acquired by vm 539cc03c-a3a8-4e56-9330-2f729667dde7
>Thread-2163::DEBUG::2014-01-27 10:25:22,362::vm::4615::vm.Vm::(destroy) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::destroy Called
>Thread-2163::DEBUG::2014-01-27 10:25:22,362::vm::4609::vm.Vm::(deleteVm) vmId=`539cc03c-a3a8-4e56-9330-2f729667dde7`::Total desktops after destroy of 539cc03c-a3a8-4e56-9330-2f729667dde7 is 0
>Thread-2163::DEBUG::2014-01-27 10:25:22,362::BindingXMLRPC::977::vds::(wrapper) return vmDestroy with {'status': {'message': 'Machine destroyed', 'code': 0}}
>VM Channels Listener::DEBUG::2014-01-27 10:25:22,613::vmChannels::112::vds::(_do_del_channels) fileno 65 was removed from listener.
>Thread-2164::DEBUG::2014-01-27 10:25:22,837::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2164::DEBUG::2014-01-27 10:25:22,868::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-39::DEBUG::2014-01-27 10:25:23,344::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:25:23,352::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.000670058 s, 524 kB/s\n'; <rc> = 0
>Thread-2165::DEBUG::2014-01-27 10:25:23,383::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call tasksList with () {}
>Thread-2165::ERROR::2014-01-27 10:25:23,403::BindingXMLRPC::986::vds::(wrapper) vdsm exception occured
>Traceback (most recent call last):
>  File "/usr/share/vdsm/BindingXMLRPC.py", line 973, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 54, in wrapper
>    rv = func(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 306, in tasksList
>    status = self.svdsmProxy.glusterTasksList(taskIds)
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in glusterTasksList
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
>    raise convert_to_error(kind, result)
>GlusterCmdExecFailedException: Command execution failed
>error: tasks is not a valid status option
>Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool]
>return code: 1
>Thread-36::DEBUG::2014-01-27 10:25:23,667::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:25:23,673::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.00076255 s, 466 kB/s\n'; <rc> = 0
>Thread-35::DEBUG::2014-01-27 10:25:25,284::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:25:25,291::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.000952538 s, 578 kB/s\n'; <rc> = 0
>Thread-2166::INFO::2014-01-27 10:25:25,408::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2166::INFO::2014-01-27 10:25:25,409::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'c074a6e3-87be-445e-890f-601c215ba320': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000952538', 'lastCheck': '0.1', 'valid': True}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00076255', 'lastCheck': '1.7', 'valid': True}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000670058', 'lastCheck': '2.1', 'valid': True}}
>Thread-2168::DEBUG::2014-01-27 10:25:27,907::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2168::DEBUG::2014-01-27 10:25:27,931::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2170::DEBUG::2014-01-27 10:25:29,318::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2170::INFO::2014-01-27 10:25:29,321::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2170::INFO::2014-01-27 10:25:29,332::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2171::DEBUG::2014-01-27 10:25:29,336::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2171::INFO::2014-01-27 10:25:29,336::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2171::INFO::2014-01-27 10:25:29,340::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67617030144', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-2173::DEBUG::2014-01-27 10:25:32,982::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2173::DEBUG::2014-01-27 10:25:33,006::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-39::DEBUG::2014-01-27 10:25:33,366::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:25:33,373::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.000728538 s, 482 kB/s\n'; <rc> = 0
>Thread-36::DEBUG::2014-01-27 10:25:33,690::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:25:33,697::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.000768288 s, 462 kB/s\n'; <rc> = 0
>Thread-35::DEBUG::2014-01-27 10:25:35,309::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:25:35,316::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.00100611 s, 548 kB/s\n'; <rc> = 0
>Thread-2176::DEBUG::2014-01-27 10:25:38,056::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2176::DEBUG::2014-01-27 10:25:38,080::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2177::DEBUG::2014-01-27 10:25:39,359::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2177::INFO::2014-01-27 10:25:39,360::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2177::INFO::2014-01-27 10:25:39,370::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2178::DEBUG::2014-01-27 10:25:39,374::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2178::INFO::2014-01-27 10:25:39,375::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2178::INFO::2014-01-27 10:25:39,378::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67616899072', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-2179::INFO::2014-01-27 10:25:40,481::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2179::INFO::2014-01-27 10:25:40,483::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'c074a6e3-87be-445e-890f-601c215ba320': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00100611', 'lastCheck': '5.2', 'valid': True}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000768288', 'lastCheck': '6.8', 'valid': True}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000728538', 'lastCheck': '7.1', 'valid': True}}
>Thread-2181::DEBUG::2014-01-27 10:25:43,132::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2181::DEBUG::2014-01-27 10:25:43,155::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-39::DEBUG::2014-01-27 10:25:43,385::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:25:43,391::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.000705992 s, 497 kB/s\n'; <rc> = 0
>Thread-36::DEBUG::2014-01-27 10:25:43,713::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:25:43,720::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.00089472 s, 397 kB/s\n'; <rc> = 0
>Thread-35::DEBUG::2014-01-27 10:25:45,334::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:25:45,342::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.000879242 s, 627 kB/s\n'; <rc> = 0
>Thread-2184::DEBUG::2014-01-27 10:25:48,190::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2184::DEBUG::2014-01-27 10:25:48,213::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2185::DEBUG::2014-01-27 10:25:49,399::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2185::INFO::2014-01-27 10:25:49,399::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2185::INFO::2014-01-27 10:25:49,411::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2186::DEBUG::2014-01-27 10:25:49,415::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2186::INFO::2014-01-27 10:25:49,416::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2186::INFO::2014-01-27 10:25:49,419::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67616899072', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-2189::DEBUG::2014-01-27 10:25:53,251::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {} flowID [3833025d]
>Thread-2189::DEBUG::2014-01-27 10:25:53,275::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-39::DEBUG::2014-01-27 10:25:53,404::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:25:53,411::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.000699078 s, 502 kB/s\n'; <rc> = 0
>Thread-36::DEBUG::2014-01-27 10:25:53,735::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:25:53,742::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.000890666 s, 399 kB/s\n'; <rc> = 0
>Thread-35::DEBUG::2014-01-27 10:25:55,360::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:25:55,367::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.000987287 s, 558 kB/s\n'; <rc> = 0
>Thread-2190::INFO::2014-01-27 10:25:55,554::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2190::INFO::2014-01-27 10:25:55,554::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'c074a6e3-87be-445e-890f-601c215ba320': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000987287', 'lastCheck': '0.2', 'valid': True}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000890666', 'lastCheck': '1.8', 'valid': True}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000699078', 'lastCheck': '2.1', 'valid': True}}
>Thread-2192::DEBUG::2014-01-27 10:25:58,319::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2192::DEBUG::2014-01-27 10:25:58,342::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2194::DEBUG::2014-01-27 10:25:59,441::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2194::INFO::2014-01-27 10:25:59,442::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2194::INFO::2014-01-27 10:25:59,453::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2195::DEBUG::2014-01-27 10:25:59,456::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2195::INFO::2014-01-27 10:25:59,457::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2195::INFO::2014-01-27 10:25:59,460::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67616768000', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-2197::DEBUG::2014-01-27 10:26:03,380::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2197::DEBUG::2014-01-27 10:26:03,403::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-39::DEBUG::2014-01-27 10:26:03,423::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:26:03,429::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.000379165 s, 926 kB/s\n'; <rc> = 0
>Thread-36::DEBUG::2014-01-27 10:26:03,757::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:26:03,764::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.000803571 s, 442 kB/s\n'; <rc> = 0
>Thread-35::DEBUG::2014-01-27 10:26:05,385::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:26:05,392::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.000957546 s, 575 kB/s\n'; <rc> = 0
>Thread-2200::DEBUG::2014-01-27 10:26:08,442::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2200::DEBUG::2014-01-27 10:26:08,465::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2201::DEBUG::2014-01-27 10:26:09,481::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2201::INFO::2014-01-27 10:26:09,482::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2201::INFO::2014-01-27 10:26:09,493::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2202::DEBUG::2014-01-27 10:26:09,497::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2202::INFO::2014-01-27 10:26:09,497::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2202::INFO::2014-01-27 10:26:09,501::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67616899072', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-2203::INFO::2014-01-27 10:26:10,625::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2203::INFO::2014-01-27 10:26:10,625::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'c074a6e3-87be-445e-890f-601c215ba320': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000957546', 'lastCheck': '5.2', 'valid': True}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000803571', 'lastCheck': '6.9', 'valid': True}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000379165', 'lastCheck': '7.2', 'valid': True}}
>Thread-39::DEBUG::2014-01-27 10:26:13,441::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:26:13,448::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.000752323 s, 467 kB/s\n'; <rc> = 0
>Thread-2205::DEBUG::2014-01-27 10:26:13,511::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2205::DEBUG::2014-01-27 10:26:13,534::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-36::DEBUG::2014-01-27 10:26:13,780::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:26:13,787::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.00103163 s, 344 kB/s\n'; <rc> = 0
>Thread-35::DEBUG::2014-01-27 10:26:15,411::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:26:15,418::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.000896953 s, 614 kB/s\n'; <rc> = 0
>Thread-2208::DEBUG::2014-01-27 10:26:18,587::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2208::DEBUG::2014-01-27 10:26:18,620::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2209::DEBUG::2014-01-27 10:26:19,521::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2209::INFO::2014-01-27 10:26:19,522::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2209::INFO::2014-01-27 10:26:19,533::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2210::DEBUG::2014-01-27 10:26:19,537::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2210::INFO::2014-01-27 10:26:19,537::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2210::INFO::2014-01-27 10:26:19,541::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67616899072', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-2213::DEBUG::2014-01-27 10:26:23,421::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call tasksList with () {}
>Thread-2213::ERROR::2014-01-27 10:26:23,435::BindingXMLRPC::986::vds::(wrapper) vdsm exception occured
>Traceback (most recent call last):
>  File "/usr/share/vdsm/BindingXMLRPC.py", line 973, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 54, in wrapper
>    rv = func(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 306, in tasksList
>    status = self.svdsmProxy.glusterTasksList(taskIds)
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in glusterTasksList
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
>    raise convert_to_error(kind, result)
>GlusterCmdExecFailedException: Command execution failed
>error: tasks is not a valid status option
>Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool]
>return code: 1
>Thread-39::DEBUG::2014-01-27 10:26:23,460::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:26:23,467::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.000627245 s, 560 kB/s\n'; <rc> = 0
>Thread-2214::DEBUG::2014-01-27 10:26:23,656::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2214::DEBUG::2014-01-27 10:26:23,679::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-36::DEBUG::2014-01-27 10:26:23,802::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:26:23,809::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.000950069 s, 374 kB/s\n'; <rc> = 0
>Thread-35::DEBUG::2014-01-27 10:26:25,437::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:26:25,444::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.000902642 s, 610 kB/s\n'; <rc> = 0
>Thread-2215::INFO::2014-01-27 10:26:25,697::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2215::INFO::2014-01-27 10:26:25,697::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'c074a6e3-87be-445e-890f-601c215ba320': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000902642', 'lastCheck': '0.2', 'valid': True}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000950069', 'lastCheck': '1.9', 'valid': True}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000627245', 'lastCheck': '2.2', 'valid': True}}
>Thread-2217::DEBUG::2014-01-27 10:26:28,725::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2217::DEBUG::2014-01-27 10:26:28,748::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2219::DEBUG::2014-01-27 10:26:29,562::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2219::INFO::2014-01-27 10:26:29,563::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2219::INFO::2014-01-27 10:26:29,574::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2220::DEBUG::2014-01-27 10:26:29,577::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2220::INFO::2014-01-27 10:26:29,578::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2220::INFO::2014-01-27 10:26:29,582::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67616768000', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-39::DEBUG::2014-01-27 10:26:33,481::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:26:33,488::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.000748295 s, 469 kB/s\n'; <rc> = 0
>Thread-2222::DEBUG::2014-01-27 10:26:33,797::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2222::DEBUG::2014-01-27 10:26:33,820::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-36::DEBUG::2014-01-27 10:26:33,835::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:26:33,841::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.000653321 s, 543 kB/s\n'; <rc> = 0
>Thread-35::DEBUG::2014-01-27 10:26:35,461::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:26:35,468::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.000880031 s, 626 kB/s\n'; <rc> = 0
>Thread-2225::DEBUG::2014-01-27 10:26:38,874::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2225::DEBUG::2014-01-27 10:26:38,897::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2226::DEBUG::2014-01-27 10:26:39,603::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2226::INFO::2014-01-27 10:26:39,604::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2226::INFO::2014-01-27 10:26:39,613::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2227::DEBUG::2014-01-27 10:26:39,617::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2227::INFO::2014-01-27 10:26:39,618::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2227::INFO::2014-01-27 10:26:39,623::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67616768000', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-2228::INFO::2014-01-27 10:26:40,769::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2228::INFO::2014-01-27 10:26:40,769::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'c074a6e3-87be-445e-890f-601c215ba320': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000880031', 'lastCheck': '5.3', 'valid': True}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000653321', 'lastCheck': '6.9', 'valid': True}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000748295', 'lastCheck': '7.3', 'valid': True}}
>Thread-39::DEBUG::2014-01-27 10:26:43,500::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:26:43,507::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.000647216 s, 542 kB/s\n'; <rc> = 0
>Thread-36::DEBUG::2014-01-27 10:26:43,857::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:26:43,863::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.000846593 s, 419 kB/s\n'; <rc> = 0
>Thread-2231::DEBUG::2014-01-27 10:26:43,943::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2231::DEBUG::2014-01-27 10:26:43,966::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-35::DEBUG::2014-01-27 10:26:45,487::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:26:45,494::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.000971697 s, 567 kB/s\n'; <rc> = 0
>Thread-2233::DEBUG::2014-01-27 10:26:49,005::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2233::DEBUG::2014-01-27 10:26:49,028::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2234::DEBUG::2014-01-27 10:26:49,645::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2234::INFO::2014-01-27 10:26:49,645::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2234::INFO::2014-01-27 10:26:49,654::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2235::DEBUG::2014-01-27 10:26:49,657::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2235::INFO::2014-01-27 10:26:49,658::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2235::INFO::2014-01-27 10:26:49,661::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67616768000', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-39::DEBUG::2014-01-27 10:26:53,519::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:26:53,526::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.000721972 s, 486 kB/s\n'; <rc> = 0
>Thread-36::DEBUG::2014-01-27 10:26:53,879::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:26:53,885::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.000939488 s, 378 kB/s\n'; <rc> = 0
>Thread-2238::DEBUG::2014-01-27 10:26:54,072::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2238::DEBUG::2014-01-27 10:26:54,095::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-35::DEBUG::2014-01-27 10:26:55,514::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:26:55,521::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.00107179 s, 514 kB/s\n'; <rc> = 0
>Thread-2239::INFO::2014-01-27 10:26:55,847::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2239::INFO::2014-01-27 10:26:55,848::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'c074a6e3-87be-445e-890f-601c215ba320': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00107179', 'lastCheck': '0.3', 'valid': True}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000939488', 'lastCheck': '2.0', 'valid': True}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000721972', 'lastCheck': '2.3', 'valid': True}}
>Thread-2242::DEBUG::2014-01-27 10:26:59,143::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2242::DEBUG::2014-01-27 10:26:59,166::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2243::DEBUG::2014-01-27 10:26:59,680::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2243::INFO::2014-01-27 10:26:59,681::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2243::INFO::2014-01-27 10:26:59,691::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2244::DEBUG::2014-01-27 10:26:59,695::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2244::INFO::2014-01-27 10:26:59,696::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2244::INFO::2014-01-27 10:26:59,699::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67616768000', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-39::DEBUG::2014-01-27 10:27:03,540::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:27:03,547::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.000559236 s, 628 kB/s\n'; <rc> = 0
>Thread-36::DEBUG::2014-01-27 10:27:03,901::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:27:03,908::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.000876883 s, 405 kB/s\n'; <rc> = 0
>Thread-2246::DEBUG::2014-01-27 10:27:04,204::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2246::DEBUG::2014-01-27 10:27:04,227::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-35::DEBUG::2014-01-27 10:27:05,539::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:27:05,546::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.000928934 s, 593 kB/s\n'; <rc> = 0
>Thread-2249::DEBUG::2014-01-27 10:27:09,279::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2249::DEBUG::2014-01-27 10:27:09,302::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2250::DEBUG::2014-01-27 10:27:09,722::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2250::INFO::2014-01-27 10:27:09,722::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2250::INFO::2014-01-27 10:27:09,732::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2251::DEBUG::2014-01-27 10:27:09,736::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2251::INFO::2014-01-27 10:27:09,736::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2251::INFO::2014-01-27 10:27:09,740::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67616768000', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-2252::INFO::2014-01-27 10:27:10,921::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2252::INFO::2014-01-27 10:27:10,921::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'c074a6e3-87be-445e-890f-601c215ba320': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000928934', 'lastCheck': '5.4', 'valid': True}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000876883', 'lastCheck': '7.0', 'valid': True}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000559236', 'lastCheck': '7.4', 'valid': True}}
>Thread-39::DEBUG::2014-01-27 10:27:13,558::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-01-27 10:27:13,565::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n351 bytes (351 B) copied, 0.00059059 s, 594 kB/s\n'; <rc> = 0
>Thread-36::DEBUG::2014-01-27 10:27:13,924::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:fast__data/d5a8d1ae-cfd4-483e-bdeb-9649a1134796/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-36::DEBUG::2014-01-27 10:27:13,930::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n355 bytes (355 B) copied, 0.000929699 s, 382 kB/s\n'; <rc> = 0
>Thread-2255::DEBUG::2014-01-27 10:27:14,337::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2255::DEBUG::2014-01-27 10:27:14,360::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-35::DEBUG::2014-01-27 10:27:15,566::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/cauldron.arclab:_data/c074a6e3-87be-445e-890f-601c215ba320/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-35::DEBUG::2014-01-27 10:27:15,573::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n551 bytes (551 B) copied, 0.000983726 s, 560 kB/s\n'; <rc> = 0
>Thread-2257::DEBUG::2014-01-27 10:27:19,403::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumesList with () {}
>Thread-2257::DEBUG::2014-01-27 10:27:19,427::BindingXMLRPC::977::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'fast_data': {'transportType': ['TCP'], 'uuid': '09ae99f1-01f9-47ff-baf1-4a4999c6aac0', 'bricks': ['cauldron.arclab:/data2'], 'volumeName': 'fast_data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'test': {'transportType': ['TCP'], 'uuid': 'e2821e53-f29a-42b5-ad31-50d8066b2318', 'bricks': ['cauldron.arclab:/data1/test'], 'volumeName': 'test', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'user.cifs': 'enable', 'server.allow-insecure': 'on', 'nfs.disable': 'on', 'auth.allow': '*'}}, 'ISO_DOMAIN': {'transportType': ['TCP'], 'uuid': '3e05e788-4fbc-438d-98c0-fddfe5ab9551', 'bricks': ['cauldron.arclab:/data1/iso'], 'volumeName': 'ISO_DOMAIN', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}, 'data': {'transportType': ['TCP'], 'uuid': '26a05930-9207-4e92-9cf9-bb292eff940b', 'bricks': ['cauldron.arclab:/data'], 'volumeName': 'data', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '1', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [], 'options': {'cluster.server-quorum-type': 'server', 'cluster.eager-lock': 'enable', 'performance.stat-prefetch': 'off', 'auth.allow': '*', 'cluster.quorum-type': 'auto', 'performance.quick-read': 'off', 'network.remote-dio': 'enable', 'nfs.disable': 'off', 'performance.io-cache': 'off', 'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'user.cifs': 'enable', 'performance.read-ahead': 'off', 'storage.owner-gid': '36'}}}}
>Thread-2258::DEBUG::2014-01-27 10:27:19,761::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2258::INFO::2014-01-27 10:27:19,762::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2258::INFO::2014-01-27 10:27:19,773::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 4L}}
>Thread-2259::DEBUG::2014-01-27 10:27:19,778::BindingXMLRPC::159::vds::(wrapper) client [192.168.1.107]
>Thread-2259::INFO::2014-01-27 10:27:19,778::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a13a367c-accd-451a-a2f3-4c83660f1719', options=None)
>Thread-2259::INFO::2014-01-27 10:27:19,782::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'arclab', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 0, 'domains': 'c074a6e3-87be-445e-890f-601c215ba320:Active,d5a8d1ae-cfd4-483e-bdeb-9649a1134796:Active,abf4ff41-eb0a-4580-9ad0-bae53d56c6b0:Active', 'master_uuid': 'c074a6e3-87be-445e-890f-601c215ba320', 'version': '3', 'spm_id': 1, 'type': 'GLUSTERFS', 'master_ver': 1}, 'dominfo': {'c074a6e3-87be-445e-890f-601c215ba320': {'status': 'Active', 'diskfree': '425801940992', 'isoprefix': '', 'alerts': [], 'disktotal': '429152665600', 'version': 3}, 'd5a8d1ae-cfd4-483e-bdeb-9649a1134796': {'status': 'Active', 'diskfree': '67616768000', 'isoprefix': '', 'alerts': [], 'disktotal': '111443574784', 'version': 3}, 'abf4ff41-eb0a-4580-9ad0-bae53d56c6b0': {'status': 'Active', 'diskfree': '559528476672', 'isoprefix': '/rhev/data-center/mnt/cauldron.arclab:_ISO__DOMAIN/abf4ff41-eb0a-4580-9ad0-bae53d56c6b0/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '570291519488', 'version': 0}}}
>Thread-2262::DEBUG::2014-01-27 10:27:23,067::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumeStatus with ('data', '', '') {}
>Thread-2262::DEBUG::2014-01-27 10:27:23,086::BindingXMLRPC::977::vds::(wrapper) return volumeStatus with {'volumeStatus': {'bricks': [{'status': 'ONLINE', 'brick': 'cauldron.arclab:/data', 'pid': '1524', 'port': '49159'}], 'nfs': [{'status': 'ONLINE', 'hostname': '192.168.1.107', 'pid': '1560', 'port': '2049'}], 'shd': [], 'name': 'data'}, 'status': {'message': 'Done', 'code': 0}}
>Thread-2263::DEBUG::2014-01-27 10:27:23,096::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumeStatus with ('fast_data', '', '') {}
>Thread-2263::DEBUG::2014-01-27 10:27:23,113::BindingXMLRPC::977::vds::(wrapper) return volumeStatus with {'volumeStatus': {'bricks': [{'status': 'ONLINE', 'brick': 'cauldron.arclab:/data2', 'pid': '1516', 'port': '49160'}], 'nfs': [{'status': 'ONLINE', 'hostname': '192.168.1.107', 'pid': '1560', 'port': '2049'}], 'shd': [], 'name': 'fast_data'}, 'status': {'message': 'Done', 'code': 0}}
>Thread-2264::DEBUG::2014-01-27 10:27:23,122::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumeStatus with ('ISO_DOMAIN', '', '') {}
>Thread-2264::DEBUG::2014-01-27 10:27:23,138::BindingXMLRPC::977::vds::(wrapper) return volumeStatus with {'volumeStatus': {'bricks': [{'status': 'ONLINE', 'brick': 'cauldron.arclab:/data1/iso', 'pid': '1540', 'port': '49158'}], 'nfs': [{'status': 'ONLINE', 'hostname': '192.168.1.107', 'pid': '1560', 'port': '2049'}], 'shd': [], 'name': 'ISO_DOMAIN'}, 'status': {'message': 'Done', 'code': 0}}
>Thread-2265::DEBUG::2014-01-27 10:27:23,148::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call volumeStatus with ('test', '', '') {}
>Thread-2265::DEBUG::2014-01-27 10:27:23,164::BindingXMLRPC::977::vds::(wrapper) return volumeStatus with {'volumeStatus': {'bricks': [{'status': 'ONLINE', 'brick': 'cauldron.arclab:/data1/test', 'pid': '1530', 'port': '49161'}], 'nfs': [], 'shd': [], 'name': 'test'}, 'status': {'message': 'Done', 'code': 0}}
>Thread-2266::DEBUG::2014-01-27 10:27:23,452::BindingXMLRPC::970::vds::(wrapper) client [192.168.1.107]::call tasksList with () {}
>Thread-2266::ERROR::2014-01-27 10:27:23,466::BindingXMLRPC::986::vds::(wrapper) vdsm exception occured
>Traceback (most recent call last):
>  File "/usr/share/vdsm/BindingXMLRPC.py", line 973, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 54, in wrapper
>    rv = func(*args, **kwargs)
>  File "/usr/share/vdsm/gluster/api.py", line 306, in tasksList
>    status = self.svdsmProxy.glusterTasksList(taskIds)
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in glusterTasksList
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
>    raise convert_to_error(kind, result)
>GlusterCmdExecFailedException: Command execution failed
>error: tasks is not a valid status option
>Usage: volume status [all | <VOLNAME> [nfs|shd|<BRICK>]] [detail|clients|mem|inode|fd|callpool]
>return code: 1

Actions: View

Attachments on bug 1058253: 856008 | 856012 | 856014